🎯

Feature Selection in Machine Learning

More features rarely means a better model. Keep the signal, drop the noise — with filter, wrapper and embedded methods that make models leaner, faster and fairer.

Filter Wrapper Embedded Pipelines

Press Next → or use ← → arrow keys

Section 01

Why Feature Selection Is Critical

🧑‍💼 REAL STORY · 61% → 79% TEST

The HR model that memorised employee IDs

An attrition model with 87 features scored 94% on train but only 61% on test — textbook overfitting. Feature importance exposed the culprit: the top "predictor" was employee_id, which the model had simply memorised. Dropping 52 junk features cut train accuracy to 82% but lifted test accuracy to 79% — real generalisation.

🛡️Less overfitting

🎯Higher accuracy

⚡Faster training

🔎More interpretable

📉

The curse of dimensionality

Every feature you add expands the space exponentially, so the same data grows sparser and patterns get harder to find. Selection removes noise, not information.

Section 01 · The Problem

The Same Data, Emptier Space

🧮

Why "more features" backfires

As dimensions grow, points drift apart, distances lose meaning, and models need exponentially more data to learn the same pattern. Fewer, sharper features beat many noisy ones almost every time.

Section 02

The Three Families of Selection

🔎 Filter

stats, no model

Rank features by a statistical score — correlation, χ², mutual information, variance.

+ blazing fast, model-agnostic

− blind to feature interactions

🔁 Wrapper

train per subset

Search feature subsets by actually training the model — RFE, RFECV, forward/backward.

+ captures interactions, model-optimal

− slow, can overfit the validation set

🧬 Embedded

during training

Selection happens inside the fit — Lasso zeroing coefficients, tree importance.

+ no extra passes, handles interactions

− tied to one model type

🪜

Use them as a staircase

Filter first to sweep out obvious junk in milliseconds, then let an embedded or wrapper method do the careful, interaction-aware final cut on what remains.

Section 03 · Filter

Filter Step 1 — Drop Junk & Twins

# 1 · kill zero/near-constant variance
from sklearn.feature_selection import \
    VarianceThreshold
sel = VarianceThreshold(threshold=0.01)
X = sel.fit_transform(X_train)

# 2 · drop one of each correlated pair
corr = X_train.corr().abs()
upper = corr.where(np.triu(
    np.ones(corr.shape, bool), k=1))
drop = [c for c in upper
        if upper[c].max() > 0.90]
X_train = X_train.drop(columns=drop)

Section 03 · Filter

Filter Step 2 — Rank & Keep the Top-k

from sklearn.feature_selection import SelectKBest, f_classif, mutual_info_classif
sel = SelectKBest(score_func=f_classif, k=8)          # ANOVA F for classification
X_sel = sel.fit_transform(X_train, y_train)
# mutual_info_classif catches non-linear links; f_regression for continuous targets

Section 04

Which Statistical Test? Match the Types

A marketing team ran Pearson correlation on category codes (tier 1/2/3, region 1/2/3/4) and "found" r = 0.38 — statistically meaningless, since those integers have no order. A proper χ² test showed no association (p = 0.42).

Section 04 · Reference

Statistical Test Cheat-Sheet

Test	Feature	Target	Non-linear?	sklearn	Output
Chi-Square	categorical	categorical	no	`chi2`	χ² + p
Pearson	numeric	numeric	no	`f_regression`	r (−1…1)
Mutual Info	any	any	yes	`mutual_info_*`	bits (≥0)
ANOVA	numeric	categorical	no	`f_classif`	F + p

🔬

Mutual information is the safe generalist

When a relationship might be non-linear (age → churn rises then falls), Pearson reads ≈ 0 and hides it. Mutual information measures any dependence, so it catches signals the linear tests miss.

Section 05 · Wrapper

Wrapper Methods — RFE & RFECV

from sklearn.feature_selection import RFECV
sel = RFECV(RandomForestClassifier(), cv=5, scoring='roc_auc', min_features_to_select=3)
sel.fit(X_train, y_train);  sel.n_features_    # CV picks the optimal k — no guessing

Section 06 · Embedded

Embedded — Lasso Zeroes Weak Features

from sklearn.feature_selection import SelectFromModel
from sklearn.linear_model import LassoCV
sel = SelectFromModel(LassoCV(cv=5))       # keeps only non-zero-coefficient features
X_sel = sel.fit_transform(X_train, y_train)   # fast, interpretable, principled

Section 06 · Trust

Permutation Importance — the Honest Test

🎭

Why not just trust tree importance?

Built-in importance inflates high-cardinality columns like IDs. Permutation importance measures the real score drop when a feature is shuffled — a wide error bar around zero, like employee_id here, unmasks a memorised non-signal.

Section 08 · Results

87 → 12 Features: The Payoff

Before · 87 feats

Train acc	94.2%
Test acc	61.3%
Test AUC	0.67
Train time	48.3s
Overfit gap	32.9%

After · 12 (RFECV)

Train acc	82.1%
Test acc	79.4%
Test AUC	0.89
Train time	4.8s
Overfit gap	2.7%

🏆

Trading train accuracy for test accuracy

Train fell 12 points, test rose 18, the overfit gap collapsed from 33% to under 3%, and training got 10× faster. That trade — worse on train, better on test — is generalisation.

Section 09

One Pipeline: Prep → Select → Model

# Option A — embedded (fast)
lasso_pipe = Pipeline([
  ('prep',   preprocessor),
  ('select', SelectFromModel(LassoCV(cv=5))),
  ('model',  LogisticRegression())])

# Option B — wrapper (most accurate)
rfecv_pipe = Pipeline([
  ('prep',   preprocessor),
  ('select', RFECV(RandomForestClassifier(),
                  cv=5, scoring='roc_auc')),
  ('model',  RandomForestClassifier())])

lasso_pipe.fit(X_train, y_train)
joblib.dump(lasso_pipe, 'fs_pipeline.pkl')

Pro · Choose

Filter vs Wrapper vs Embedded — Pick One

Family	Speed	Interactions	Overfit risk	Reach for it when…
Filter	fastest	ignored	low	a first sweep over hundreds of features
Embedded (Lasso)	fast	some	low	you want speed + interpretable zeros
Wrapper (RFECV)	slow	full	medium	accuracy matters more than time

🧭

The pragmatic recipe

Filter to cut the obvious junk fast → Lasso for a quick principled shortlist → RFECV for the final, cross-validated cut when the stakes are high. Compare on test AUC, never train.

Pro · Debug

Common Pitfalls & Their Fixes

Symptom	The trap	The fix
Great CV, awful in prod	Select on the full dataset	Select inside CV / on train only
An ID tops importance	Trusting tree importance	Permutation importance
"Correlation" on categories	Pearson on integer codes	Chi-Square test
Non-linear signal missed	Only Pearson / f_test	Mutual information
Guessing k	Hand-tuning RFE's k	RFECV picks k by CV
Redundant twins kept	Skipping correlation filter	Drop one of each >0.9 pair
Wrong features at inference	Saved model, not selector	`joblib.dump(pipeline)`

Section 10 · Part 1

The Golden Rules — 1 to 4

🎯 FEATURE SELECTION · RULES 1–4

1

Select before training, never after — and inside cross-validation, so noise never contaminates the fit or leaks into the score.

2

Start with VarianceThreshold + correlation. They delete obvious junk in milliseconds before any expensive wrapper runs.

3

Match the test to the types. χ² for categorical, Pearson for linear-numeric, mutual info for non-linear, ANOVA for numeric→class.

4

Beware circular importance. Don't select features with importances from a model trained on the same full data — use permutation importance or a held-out split.

Section 10 · Part 2

The Golden Rules — 5 to 8 & Takeaway

🎯 FEATURE SELECTION · RULES 5–8

5

Prefer RFECV over RFE — cross-validation finds the optimal feature count so you never hand-guess k.

6

Reach for Lasso when you have many features and want fast, interpretable selection — its zero-coefficients are mathematically principled.

7

Compare on test, not train. High train accuracy with weak test accuracy is overfitting wearing a disguise.

8

Save the whole pipeline — preprocessing + selection + model as one joblib object, or inference selects the wrong features.

🎯

Selection removes noise, not data

Cutting the HR model from 87 features to 12 lowered train accuracy 12 points but raised test accuracy 18 — worse on paper, far better in reality. Trading memorisation for generalisation is the whole point of machine learning.

🎯 End of tutorial · Press ← to review, or click Restart