Feature Selection in Machine Learning

Why Feature Selection Matters 🎯Feature Selection क्यों ज़रूरी है 🎯为什么特征选择很重要 🎯

More columns do not automatically mean a better model. A column that carries no signal still costs something: the model may fit its random ups and downs (overfitting), distance-based models treat it as just as important as real signals, and every extra column adds training time. With many columns and few rows, the data also becomes thin — the curse of dimensionality. Feature selection keeps the columns that help and drops the rest.

ज़्यादा columns का मतलब अपने-आप बेहतर model नहीं। जिस column में कोई संकेत नहीं, उसकी भी क़ीमत है: model उसके random उतार-चढ़ाव को fit कर सकता है (overfitting), दूरी-आधारित models उसे असली संकेतों जितना ही अहम मानते हैं, और हर extra column training समय बढ़ाता है। बहुत columns और कम rows में data पतला हो जाता है — curse of dimensionality। Feature selection काम के columns रखता है और बाक़ी हटाता है।

列越多并不自动意味着模型越好。没有信号的列也有代价：模型可能去拟合它的随机起伏（过拟合），基于距离的模型会把它看得和真实信号一样重要，而且每多一列都会增加训练时间。列多行少时，数据还会变得稀疏 —— 这就是维度灾难。特征选择保留有用的列，去掉其余的。

# 1) Same models, all 36 columns vs only the useful ones
import numpy as np, pandas as pd, time
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import make_pipeline
from sklearn.linear_model import LogisticRegression
from sklearn.neighbors import KNeighborsClassifier
from sklearn.ensemble import RandomForestClassifier
from sklearn.metrics import roc_auc_score
df = pd.read_csv("churn.csv")
X = pd.get_dummies(df.drop(columns="churn"), columns=["region", "country"], dtype=int)
y = df["churn"]
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25, random_state=0, stratify=y)
useful = ["tenure", "monthly_charges", "credit_score", "num_products", "age", "income", "expenses", "data_gb",
          "region_East", "region_North", "region_South", "region_West"]
models = {"Logistic": make_pipeline(StandardScaler(), LogisticRegression(max_iter=2000)),
          "KNN (k=25)": make_pipeline(StandardScaler(), KNeighborsClassifier(25)),
          "Random forest": RandomForestClassifier(300, min_samples_leaf=3, random_state=0, n_jobs=-1)}
print(f"{'model':<14}{'columns':>8}{'train AUC':>11}{'test AUC':>10}")
for name, m in models.items():
    for cols in [list(X.columns), useful]:
        m.fit(X_train[cols], y_train)
        tr = roc_auc_score(y_train, m.predict_proba(X_train[cols])[:, 1])
        te = roc_auc_score(y_test, m.predict_proba(X_test[cols])[:, 1])
        print(f"{name:<14}{len(cols):>8}{tr:>11.3f}{te:>10.3f}")

The Three Families 🗺️तीन परिवार 🗺️三大类方法 🗺️

Filter Methods — Cheap First Checks 🔵Filter Methods — सस्ती पहली जाँच 🔵过滤法 —— 低成本的第一步 🔵

Step 1 — Constant and near-constant columnsStep 1 — Constant और लगभग constant columns第 1 步 —— 常数列和近常数列

A column that never changes cannot help any model. VarianceThreshold removes columns whose variance is at or below a threshold. Two cautions: variance depends on units (a threshold of 0.01 means something very different for rupees and for a 0/1 flag), and it must run before scaling — after StandardScaler every column has variance 1.

जो column कभी नहीं बदलता वह किसी model की मदद नहीं कर सकता। VarianceThreshold उन columns को हटाता है जिनका variance threshold के बराबर या कम है। दो सावधानियाँ: variance इकाइयों पर निर्भर है (0.01 threshold रुपयों और 0/1 flag के लिए बहुत अलग है), और इसे scaling से पहले चलाना ज़रूरी है — StandardScaler के बाद हर column का variance 1 होता है।

从不变化的列对任何模型都没有帮助。VarianceThreshold 会删除方差不超过阈值的列。两点注意：方差取决于单位（阈值 0.01 对卢比和 0/1 标记意义完全不同）；而且必须在缩放之前运行 —— StandardScaler 之后每列方差都是 1。

Step 2 — Near-duplicate columnsStep 2 — लगभग duplicate columns第 2 步 —— 近似重复的列

Two columns with |r| above about 0.9 carry almost the same information. Keep one. The original code dropped whichever column came later in the table; the version below keeps the one more related to the target. A high correlation is not always redundancy, though: income and expenses correlate at 0.87 here, and both are needed.

|r| लगभग 0.9 से ऊपर वाले दो columns लगभग एक ही जानकारी रखते हैं। एक रखें। मूल code table में बाद में आने वाला column हटा देता था; नीचे का version target से ज़्यादा जुड़ा column रखता है। फिर भी ऊँचा correlation हमेशा दोहराव नहीं: यहाँ income और expenses का correlation 0.87 है, और दोनों चाहिए।

|r| 超过约 0.9 的两列几乎携带相同信息，只留一列。原代码删除的是表中靠后的那一列；下面的版本保留与目标关系更强的那一列。不过高相关并不总是冗余：这里收入和支出的相关系数为 0.87，而两者都需要。

# 2) Cheap filters: constant columns, then near-duplicate columns
from sklearn.feature_selection import VarianceThreshold
vt = VarianceThreshold(threshold=0.01).fit(X_train)                  # variance depends on units — check what it removes
print("removed by VarianceThreshold:", X_train.columns[~vt.get_support()].tolist())
X_tr = X_train.loc[:, vt.get_support()]
corr = X_tr.corr().abs()
upper = corr.where(np.triu(np.ones(corr.shape, dtype=bool), k=1))
target_r = X_tr.corrwith(y_train).abs()
to_drop = set()
for a in upper.columns:
    for b in upper.index[upper[a] > 0.90]:                               # pair (b, a) is highly correlated
        weaker = a if target_r[a] < target_r[b] else b                   # keep the one closer to the target
        to_drop.add(weaker)
        print(f"{b} ↔ {a}: r = {corr.loc[b, a]:.3f} | |r| with churn {target_r[b]:.4f} vs {target_r[a]:.4f} → drop {weaker}")
print("columns left:", X_tr.shape[1] - len(to_drop))

Step 3 — One score per columnStep 3 — हर column का एक score第 3 步 —— 每列一个分数

SelectKBest scores each column against the target and keeps the top k. f_classif (ANOVA F) asks "do the class means differ?". mutual_info_classif asks "does knowing this column reduce uncertainty about the target in any way?" — it is reported in nats (natural log), not bits, and it is an estimate that wobbles with the random seed and the sample.

SelectKBest हर column को target के सामने score करता है और top k रखता है। f_classif (ANOVA F) पूछता है "क्या classes के means अलग हैं?"। mutual_info_classif पूछता है "क्या यह column जानने से target की अनिश्चितता किसी भी तरह घटती है?" — इसे nats (natural log) में बताया जाता है, bits में नहीं, और यह एक अनुमान है जो random seed और sample के साथ हिलता है।

SelectKBest 将每列与目标比较打分，保留前 k 列。f_classif（ANOVA F）问的是“各类别的均值是否不同？”。mutual_info_classif 问的是“知道这一列能否以任何方式减少对目标的不确定性？”—— 单位是 nats（自然对数），不是 bits，而且它是一个估计值，会随随机种子和样本波动。

# 3) Univariate scores: ANOVA F-test and mutual information (both look at ONE column at a time)
from sklearn.feature_selection import SelectKBest, f_classif, mutual_info_classif
num = [c for c in X_tr.columns if c not in to_drop]
F, p = f_classif(X_train[num], y_train)
mi = mutual_info_classif(X_train[num], y_train, discrete_features=[X_train[c].nunique() <= 5 for c in num], random_state=0)
scores = pd.DataFrame({"F": F, "p_value": p, "MI (nats)": mi}, index=num).sort_values("F", ascending=False)
scores["F rank"] = range(1, len(num) + 1)
scores["MI rank"] = scores["MI (nats)"].rank(ascending=False, method="first").astype(int)
print(scores.round(4).head(12).to_string())
print("...\nwhere are the tricky ones?\n", scores.loc[["age", "income", "customer_id"]].round(4).to_string())
kbest = SelectKBest(f_classif, k=8).fit(X_train[num], y_train)
print("\nSelectKBest(f_classif, k=8) keeps:", kbest.get_feature_names_out().tolist())

Chi-Square vs Correlation 🧮Chi-Square बनाम Correlation 🧮卡方检验 vs 相关系数 🧮

The right test depends on the types of the two variables. Pearson correlation measures a straight-line relationship between two numbers. Chi-square tests whether two categorical variables are independent. The classic mistake is to turn categories into codes (North = 1, South = 2, …) and run Pearson — the answer then depends on how you numbered them.

सही test दोनों variables के प्रकार पर निर्भर है। Pearson correlation दो संख्याओं के बीच सीधी-रेखा संबंध मापता है। Chi-square जाँचता है कि दो categorical variables स्वतंत्र हैं या नहीं। आम ग़लती है categories को codes (North = 1, South = 2, …) बनाकर Pearson चलाना — तब जवाब इस पर निर्भर करता है कि आपने उन्हें कैसे नंबर दिया।

选择哪种检验取决于两个变量的类型。Pearson 相关衡量两个数值之间的直线关系。卡方检验判断两个类别变量是否独立。经典错误是把类别变成编码（North = 1、South = 2……）再算 Pearson —— 结果就取决于你怎么编号。

# 4) A categorical column: chi-square vs Pearson on label codes
from scipy.stats import chi2_contingency, pearsonr
train = df.loc[X_train.index]
table = pd.crosstab(train["region"], train["churn"])
chi2_stat, p_val, dof, _ = chi2_contingency(table)
cramers_v = np.sqrt(chi2_stat / (table.values.sum() * (min(table.shape) - 1)))
print(table.assign(churn_rate=(table[1] / table.sum(axis=1)).round(3)))
print(f"\nchi-square = {chi2_stat:.1f}, dof = {dof}, p = {p_val:.1e}, Cramér's V = {cramers_v:.3f}")
for order in [["North", "South", "East", "West"], ["East", "North", "West", "South"], ["South", "East", "West", "North"]]:
    codes = train["region"].map({r: i + 1 for i, r in enumerate(order)})
    r, p = pearsonr(codes, train["churn"])
    print("codes " + ", ".join(f"{g}={i + 1}" for i, g in enumerate(order)) + f" → Pearson r = {r:+.3f}, p = {p:.3f}")

Density Plots — Look Before You Test 👀Density Plots — Test से पहले देखें 👀密度图 —— 检验之前先看 👀

Plot each column's distribution separately for each class. If the curves sit apart, the column separates the classes. If they lie on top of each other, the column is probably useless on its own. A one-line version: sns.kdeplot(data=train, x="age", hue="churn", common_norm=False) — use training rows only.

हर column का distribution हर class के लिए अलग से plot करें। अगर curves अलग हों, तो column classes को अलग करता है। अगर एक-दूसरे पर हों, तो column शायद अकेले बेकार है। एक line में: sns.kdeplot(data=train, x="age", hue="churn", common_norm=False) — सिर्फ़ training rows इस्तेमाल करें।

为每个类别分别画出每一列的分布。曲线分开，说明该列能区分类别；曲线重叠，说明该列单独看可能无用。一行代码：sns.kdeplot(data=train, x="age", hue="churn", common_norm=False) —— 只用训练行。

Wrapper Methods — RFE and RFECV 🟢Wrapper Methods — RFE और RFECV 🟢包装法 —— RFE 与 RFECV 🟢

RFE (recursive feature elimination) fits a model, removes the column with the smallest importance (for a linear model: the smallest |coefficient| on scaled data), and repeats. RFECV does this inside cross-validation and picks the number of columns with the best average score. Because the model sees all remaining columns together, it can keep columns that only work in combination.

RFE (recursive feature elimination) model fit करता है, सबसे कम importance वाला column हटाता है (linear model के लिए: scaled data पर सबसे छोटा |coefficient|), और दोहराता है। RFECV यह cross-validation के अंदर करता है और सबसे अच्छे औसत score वाली columns की संख्या चुनता है। क्योंकि model बचे हुए सभी columns को साथ देखता है, वह ऐसे columns रख सकता है जो सिर्फ़ मिलकर काम करते हैं।

RFE（递归特征消除）先拟合模型，删除重要性最小的列（线性模型即标准化数据上 |系数| 最小的列），然后重复。RFECV 在交叉验证中做这件事，并选出平均得分最高的列数。由于模型同时看到所有剩余列，它能保留那些只有组合起来才有用的列。

# 5) Wrapper: RFECV — fit, drop the weakest column, repeat; score every size with 5-fold CV
from sklearn.feature_selection import RFECV
from sklearn.model_selection import cross_val_score
logit = make_pipeline(StandardScaler(), LogisticRegression(max_iter=2000))
rfecv = RFECV(logit, step=1, cv=5, scoring="roc_auc", importance_getter="named_steps.logisticregression.coef_")
rfecv.fit(X_train[num], y_train)
cv_auc = rfecv.cv_results_["mean_test_score"]
print(f"best size: {rfecv.n_features_} columns → kept {rfecv.get_feature_names_out().tolist()}")
print("CV AUC by number of columns:", {k: round(float(cv_auc[k - 1]), 3) for k in [1, 2, 3, 5, 8, 10, 15, 20, len(num)]})
for cols in [["income"], ["expenses"], ["income", "expenses"]]:
    print(f"CV AUC using only {cols}: {cross_val_score(logit, X_train[cols], y_train, cv=5, scoring='roc_auc').mean():.3f}")
for name, cols in [("all columns", num), ("F-test top 8", kbest.get_feature_names_out().tolist()),
                   ("RFECV", rfecv.get_feature_names_out().tolist())]:
    logit.fit(X_train[cols], y_train)
    print(f"logistic, {name:<13} ({len(cols):>2} cols): test AUC {roc_auc_score(y_test, logit.predict_proba(X_test[cols])[:, 1]):.3f}")

Embedded Methods — L1 and Tree Importance 🟡Embedded Methods — L1 और Tree Importance 🟡嵌入法 —— L1 与树的重要性 🟡

L1 penalty (Lasso) for classificationClassification के लिए L1 penalty (Lasso)用于分类的 L1 惩罚（Lasso）

An L1 penalty pushes small coefficients to exactly zero, so fitting the model also selects columns. For a 0/1 target, use L1 logistic regression — the original page fitted LassoCV, a regression model, to the churn labels. In scikit-learn 1.8+ write LogisticRegression(l1_ratio=1.0, solver="saga"); the old penalty="l1" is deprecated. Smaller C means a stronger penalty and fewer columns. Scale first, or the penalty hits columns unevenly.

L1 penalty छोटे coefficients को ठीक शून्य पर धकेलती है, इसलिए model fit करना ही columns चुनना है। 0/1 target के लिए L1 logistic regression इस्तेमाल करें — मूल page ने churn labels पर LassoCV, एक regression model, fit किया था। scikit-learn 1.8+ में LogisticRegression(l1_ratio=1.0, solver="saga") लिखें; पुराना penalty="l1" deprecated है। छोटा C यानी मज़बूत penalty और कम columns। पहले scale करें, वरना penalty columns पर असमान पड़ती है।

L1 惩罚会把小系数压到恰好为零，因此拟合模型的同时就选出了列。对 0/1 目标，请用 L1 逻辑回归 —— 原页面把回归模型 LassoCV 拟合到流失标签上。在 scikit-learn 1.8+ 中写 LogisticRegression(l1_ratio=1.0, solver="saga")；旧的 penalty="l1" 已弃用。C 越小惩罚越强、列越少。请先缩放，否则惩罚对各列不均衡。

# 6) Embedded: L1-penalised logistic regression (the classification "Lasso")
from sklearn.linear_model import LogisticRegressionCV
from sklearn.feature_selection import SelectFromModel
cols6 = X_tr.columns.tolist()                                           # constant columns gone, BOTH data_gb and streaming_hours kept
sc6 = StandardScaler().fit(X_train[cols6])
Xs = sc6.transform(X_train[cols6])
l1cv = LogisticRegressionCV(Cs=np.logspace(-3, 1, 25), l1_ratios=[1.0], solver="saga", cv=5, scoring="roc_auc",
                            max_iter=5000, random_state=0, use_legacy_attributes=False).fit(Xs, y_train)
coef = pd.Series(l1cv.coef_[0], index=cols6)
print(f"best C = {l1cv.C_:.4f} → {int((coef != 0).sum())} non-zero of {len(cols6)}")
print(coef[coef != 0].round(3).sort_values(key=abs, ascending=False).to_string())
print("test AUC:", round(roc_auc_score(y_test, l1cv.predict_proba(sc6.transform(X_test[cols6]))[:, 1]), 3))
# which twin survives? refit on 100 bootstrap samples at the chosen C
pick = {"data_gb": 0, "streaming_hours": 0, "both": 0, "neither": 0}
r = np.random.default_rng(0)
for b in range(100):
    i = r.integers(0, len(Xs), len(Xs))
    c = LogisticRegression(C=l1cv.C_, l1_ratio=1.0, solver="saga", max_iter=5000).fit(Xs[i], y_train.to_numpy()[i]).coef_[0]
    a, s = c[cols6.index("data_gb")] != 0, c[cols6.index("streaming_hours")] != 0
    pick["both" if a and s else "data_gb" if a else "streaming_hours" if s else "neither"] += 1
print("twins kept over 100 bootstrap fits:", pick)

Random forest importance — impurity vs permutationRandom forest importance — impurity बनाम permutation随机森林重要性 —— 不纯度 vs 置换

feature_importances_ adds up how much each column reduced impurity in the training trees. It is biased toward columns with many distinct values — continuous noise and IDs offer many possible split points, and a deep tree uses them to memorise training rows. Permutation importance shuffles one column on held-out data and measures how much the score drops; useless columns score near zero.

feature_importances_ जोड़ता है कि हर column ने training trees में impurity कितनी घटाई। यह बहुत अलग-अलग मानों वाले columns की ओर पक्षपाती है — continuous शोर और IDs कई split points देते हैं, और गहरा tree उनसे training rows याद कर लेता है। Permutation importance held-out data पर एक column को shuffle करके मापता है कि score कितना गिरा; बेकार columns का score लगभग शून्य होता है।

feature_importances_ 累加每列在训练树中降低的不纯度。它偏向取值很多的列 —— 连续噪声和 ID 提供大量可切分点，深树会用它们记住训练行。置换重要性在留出数据上打乱某一列，测量得分下降多少；无用的列得分接近零。

# 7) Random forest importance: impurity (built in) vs permutation (on the test set)
from sklearn.inspection import permutation_importance
rf = RandomForestClassifier(300, random_state=0, n_jobs=-1).fit(X_train[num], y_train)
perm = permutation_importance(rf, X_test[num], y_test, scoring="roc_auc", n_repeats=20, random_state=0, n_jobs=-1)
imp = pd.DataFrame({"impurity": rf.feature_importances_, "permutation": perm.importances_mean,
                    "perm_std": perm.importances_std}, index=num)
imp["impurity rank"] = imp["impurity"].rank(ascending=False).astype(int)
imp["perm rank"] = imp["permutation"].rank(ascending=False).astype(int)
print(imp.sort_values("impurity", ascending=False).round(4).head(10).to_string())
c = imp.loc["customer_id"]
print(f"customer_id: impurity {c.impurity:.4f} (rank {int(c['impurity rank'])}) | permutation {c.permutation:+.4f} (rank {int(c['perm rank'])})")
print(f"\nimpurity says the 20 noise columns hold {imp.loc[imp.index.str.startswith('noise'), 'impurity'].sum():.0%} of the importance;"
      f" permutation gives them {imp.loc[imp.index.str.startswith('noise'), 'permutation'].sum():+.4f} AUC in total")
sfm = SelectFromModel(rf, prefit=True, threshold="mean")
print("SelectFromModel(threshold='mean') on impurity keeps:", np.array(num)[sfm.get_support()].tolist())

So SelectFromModel(rf, threshold="mean") on impurity importance kept noise_01 and noise_19, and the ID ranked 10th. The original page said the ID had "near-zero importance" — with impurity importance it does not. Use permutation importance on held-out data (or on CV folds) when you select with a forest.

इसलिए impurity importance पर SelectFromModel(rf, threshold="mean") ने noise_01 और noise_19 रखे, और ID 10वें स्थान पर था। मूल page ने कहा था कि ID की "लगभग शून्य importance" है — impurity importance में ऐसा नहीं। Forest से चुनते समय held-out data (या CV folds) पर permutation importance इस्तेमाल करें।

所以基于不纯度重要性的 SelectFromModel(rf, threshold="mean") 保留了 noise_01 和 noise_19，而 ID 排在第 10。原页面说 ID 的“重要性接近零”—— 用不纯度重要性时并非如此。用森林做选择时，请在留出数据（或交叉验证折）上使用置换重要性。

The Selection Trap — Select Inside Cross-Validation 🔒Selection का जाल — Cross-Validation के अंदर चुनें 🔒选择陷阱 —— 在交叉验证内部做选择 🔒

Feature selection looks at the target. If you choose columns using all rows and then cross-validate, the test folds have already helped pick the columns. With many candidate columns, some will match the labels by pure chance, and the score becomes fiction. The fix: put the selector inside a Pipeline so it is refitted on the training part of every fold.

Feature selection target को देखता है। अगर आप सभी rows से columns चुनकर फिर cross-validate करते हैं, तो test folds पहले ही columns चुनने में मदद कर चुके हैं। बहुत candidate columns हों तो कुछ संयोग से labels से मेल खाएँगे, और score कहानी बन जाता है। उपाय: selector को Pipeline में रखें ताकि हर fold के training हिस्से पर दोबारा fit हो।

特征选择会看目标。如果先用全部行选列再做交叉验证，测试折其实已经参与了选列。候选列很多时，总有一些会纯属偶然地与标签吻合，分数就成了虚构。解决办法：把选择器放进 Pipeline，让它在每一折的训练部分上重新拟合。

# 8) The selection trap: choose columns on ALL rows, then cross-validate → a fake score on pure noise
from sklearn.pipeline import Pipeline
def selection_gap(n_cols, seed, n_rows=100, k=20):
    r = np.random.default_rng(seed)
    Xn, yn = r.normal(size=(n_rows, n_cols)), r.integers(0, 2, n_rows)      # NO real signal anywhere
    keep = SelectKBest(f_classif, k=k).fit(Xn, yn).get_support()             # wrong: selector saw every row
    wrong = cross_val_score(LogisticRegression(max_iter=1000), Xn[:, keep], yn, cv=5).mean()
    right = cross_val_score(Pipeline([("sel", SelectKBest(f_classif, k=k)), ("lr", LogisticRegression(max_iter=1000))]),
                            Xn, yn, cv=5).mean()                             # right: selection redone inside each fold
    return wrong, right
for n_cols in [20, 100, 1000, 5000]:
    res = np.array([selection_gap(n_cols, s) for s in range(20)])
    print(f"{n_cols:>5} noise columns → top 20 kept | CV accuracy wrong {res[:, 0].mean():.3f}, right {res[:, 1].mean():.3f}")

Before vs After — All Methods Compared 📊पहले बनाम बाद — सभी Methods की तुलना 📊前后对比 —— 所有方法比较 📊

Each method chose its columns from training rows only. Then the same two models were trained and judged on the untouched test set. "the 12 true columns" is a reference you never have in real life.

हर method ने सिर्फ़ training rows से columns चुने। फिर वही दो models train हुए और अनछुए test set पर परखे गए। "the 12 true columns" एक reference है जो असल ज़िंदगी में कभी नहीं मिलता।

每种方法都只用训练行选列。然后用同样两个模型训练，并在未动过的测试集上评估。“12 个真实列”只是参考，现实中你永远拿不到。

# 9) Every method side by side — the same two models, judged on the untouched test set
from sklearn.feature_selection import SelectFromModel
mi_top10 = scores["MI (nats)"].nlargest(10).index.tolist()
l1_c = LogisticRegression(C=0.05, l1_ratio=1.0, solver="saga", max_iter=5000).fit(Xs, y_train).coef_[0]
sets = {"all columns": list(X.columns),
        "variance + correlation": num,
        "F-test top 8": kbest.get_feature_names_out().tolist(),
        "MI top 10": mi_top10,
        "RFECV (logistic)": rfecv.get_feature_names_out().tolist(),
        "L1 logistic (C = 0.05)": [c for c, w in zip(cols6, l1_c) if w != 0],
        "RF impurity > mean": np.array(num)[sfm.get_support()].tolist(),
        "the 12 true columns": useful}
rows = []
for name, cols in sets.items():
    row = {"method": name, "columns": len(cols)}
    for mname, m in [("logistic", make_pipeline(StandardScaler(), LogisticRegression(max_iter=2000))),
                     ("KNN", make_pipeline(StandardScaler(), KNeighborsClassifier(25)))]:
        m.fit(X_train[cols], y_train)
        row[f"{mname} test AUC"] = roc_auc_score(y_test, m.predict_proba(X_test[cols])[:, 1])
    rows.append(row)
bench = pd.DataFrame(rows).set_index("method")
print(bench.round(3).to_string())

A Complete Selection Pipeline 🔗पूरी Selection Pipeline 🔗完整的特征选择管道 🔗

Put every step in one Pipeline: imputation, the variance check (before scaling), scaling, encoding, selection and the model. Cross-validation then refits the selector in each fold, and the saved pipeline applies exactly the same columns to new data. Two bugs fixed from the original: pipeline.score() returns accuracy, not AUC, so we compute AUC with roc_auc_score; and new_raw_data was never defined, so we predict on a real test row.

हर क़दम एक Pipeline में रखें: imputation, variance जाँच (scaling से पहले), scaling, encoding, selection और model। तब cross-validation हर fold में selector को दोबारा fit करता है, और saved pipeline नए data पर बिल्कुल वही columns लगाती है। मूल से दो bugs ठीक किए: pipeline.score() accuracy देता है, AUC नहीं, इसलिए हम roc_auc_score से AUC निकालते हैं; और new_raw_data कभी define नहीं हुआ था, इसलिए हम एक असली test row पर predict करते हैं।

把每一步放进同一个 Pipeline：填补、方差检查（缩放前）、缩放、编码、选择和模型。这样交叉验证会在每一折重新拟合选择器，保存的管道对新数据使用完全相同的列。修正了原代码的两个错误：pipeline.score() 返回的是准确率而不是 AUC，所以我们用 roc_auc_score 计算 AUC；new_raw_data 从未定义，所以我们对一条真实测试行做预测。

# 10) A production pipeline: selection is fitted inside every CV fold and saved with the model
#     (RFECV inside cross_val_score = nested CV, so this takes a few seconds)
import joblib
from sklearn.compose import ColumnTransformer
from sklearn.preprocessing import OneHotEncoder
from sklearn.impute import SimpleImputer
from sklearn.pipeline import Pipeline
raw = df.drop(columns="churn")
R_train, R_test = raw.loc[X_train.index], raw.loc[X_test.index]
num_cols = [c for c in raw.columns if c not in ("region", "country")]
prep = ColumnTransformer([
    ("num", Pipeline([("imp", SimpleImputer(strategy="median")),
                     ("var", VarianceThreshold(0.01)),        # BEFORE scaling: after it every variance is 1
                     ("sc", StandardScaler())]), num_cols),
    ("cat", OneHotEncoder(handle_unknown="ignore", sparse_output=False), ["region", "country"])])
pipe = Pipeline([
    ("prep", prep),
    ("drop_constant", VarianceThreshold(0.0)),             # catches the one-hot column of "country" (always IN)
    ("select", RFECV(LogisticRegression(max_iter=2000), cv=5, scoring="roc_auc")),   # wrapper, refitted inside every fold
    ("model", KNeighborsClassifier(25))])
cv = cross_val_score(pipe, R_train, y_train, cv=5, scoring="roc_auc")
pipe.fit(R_train, y_train)
test_auc = roc_auc_score(y_test, pipe.predict_proba(R_test)[:, 1])      # not pipe.score(): that is accuracy
print(f"CV AUC {cv.mean():.3f} ± {cv.std():.3f} | test AUC {test_auc:.3f}")
print("columns the model uses:", pipe[:-1].get_feature_names_out().tolist())
joblib.dump(pipe, "churn_pipeline.joblib")
print("new customer → churn probability", round(pipe.predict_proba(R_test.head(1))[0, 1], 3))

RFECV inside the pipeline kept 11 columns — no noise, no ID — and lifted KNN from test AUC 0.711 (all columns) to 0.773.

Pipeline के अंदर RFECV ने 11 columns रखे — कोई शोर नहीं, कोई ID नहीं — और KNN को test AUC 0.711 (सभी columns) से 0.773 पर पहुँचाया।

管道内的 RFECV 保留了 11 列 —— 没有噪声，也没有 ID —— 把 KNN 的测试 AUC 从 0.711（全部列）提升到 0.773。

Make the Practice Data 🐍Practice Data बनाएँ 🐍生成练习数据 🐍

Run this first. Churn depends on tenure, monthly charges, credit score, number of products, region, data usage, a U-shaped age effect and the expenses ÷ income ratio. Everything else is a trap: an ID, a constant, a near-constant flag, a near-copy and 20 noise columns.

इसे पहले चलाएँ। Churn tenure, monthly charges, credit score, products की संख्या, region, data usage, U-आकार के age असर और expenses ÷ income ratio पर निर्भर है। बाक़ी सब जाल है: एक ID, एक constant, लगभग constant flag, एक लगभग copy और 20 शोर columns।

请先运行它。流失取决于在网时长、月费、信用分、产品数、地区、流量用量、U 形的年龄效应以及支出 ÷ 收入比率。其余都是陷阱：一个 ID、一个常数列、一个近常数标记、一个近似副本和 20 个噪声列。

# 0) Practice data: a telecom churn table with useful, redundant, useless and tricky columns
import numpy as np, pandas as pd
rng = np.random.default_rng(11)
n = 2000
tenure = rng.integers(1, 73, n)                                        # months as a customer
monthly_charges = np.round(rng.uniform(200, 1500, n), -1)              # ₹ per month
credit_score = np.round(rng.normal(700, 60, n))
num_products = rng.integers(1, 6, n)
age = rng.integers(18, 76, n)
log_inc = rng.normal(11, 0.5, n)
income = np.round(np.exp(log_inc), -2)                                 # ₹ per month
ratio = np.clip(rng.normal(0.65, 0.18, n), 0.15, 1.3)                  # expenses ÷ income
expenses = np.round(income * ratio, -2)
data_gb = np.round(np.clip(rng.gamma(4, 6, n), 0.5, None), 1)          # monthly data used
streaming_hours = np.round(data_gb * 1.9 + rng.normal(0, 7, n), 1)     # almost a copy of data_gb
region = rng.choice(["North", "South", "East", "West"], n)
z = (-0.035 * (tenure - 36) + 0.0016 * (monthly_charges - 850) - 0.012 * (credit_score - 700)
     - 0.25 * (num_products - 3) + 0.003 * (age - 46) ** 2 - 1.6
     + 5.0 * (ratio - 0.65) + 0.03 * (data_gb - 24)
     + pd.Series(region).map({"North": 0.0, "South": 0.9, "East": -0.5, "West": 0.6}).to_numpy())
churn = (rng.random(n) < 1 / (1 + np.exp(-z))).astype(int)
df = pd.DataFrame({"customer_id": rng.permutation(np.arange(100001, 100001 + n)), "tenure": tenure,
                   "monthly_charges": monthly_charges, "credit_score": credit_score, "num_products": num_products,
                   "age": age, "income": income, "expenses": expenses, "data_gb": data_gb,
                   "streaming_hours": streaming_hours, "region": region, "country": "IN",
                   "is_test_account": (rng.random(n) < 0.003).astype(int)})
for k in range(1, 21):                                                 # 20 columns of pure noise
    df[f"noise_{k:02d}"] = np.round(rng.normal(0, 1, n), 3)
df["churn"] = churn
df.to_csv("churn.csv", index=False)
print(f"churn.csv: {df.shape[0]:,} rows × {df.shape[1] - 1} feature columns, churn rate {churn.mean():.1%}")

Golden Rules of Feature Selection 🏆Feature Selection के सुनहरे नियम 🏆特征选择黄金法则 🏆