Machine Learning 📂 Supervised Learning · 4 of 19 208 min read

Cross-Validation in Machine Learning

A comprehensive deep-dive into every major cross-validation technique — Standard K-Fold, Stratified K-Fold, LOOCV, Group K-Fold

🌐 Language
▶️
This Tutorial Has Eight Simulators

Look for the interactive boxes: The Lucky-Split Machine (Section 01), K-Fold Rotator (03), Stratified vs Plain K-Fold (04), Group Leakage Lab (07), Time-Series Split Explorer (08), The Optimism Gap (10), Preprocessing Leakage (12) and Which Cross-Validation Should I Use? (13). Every number in them is calculated live in your browser.

Section 01

The Story That Explains Cross-Validation

📖 Real-World Analogy
The Medical Exam Board — Testing Students Fairly
A medical school wants to know if its students are truly good doctors — not just good at memorising old exam papers. If the professors ask only questions the students have already seen, everyone scores 95%. But put the same students in front of real patients, and many of them freeze.

So the exam board uses a smarter system: it rotates. In round 1, one group of students is tested on new cases while the others study. In round 2, the next group is tested. After 10 rounds, every student has been tested on cases they never studied. The board now has a fair picture of every student's real ability.

This rotating exam is cross-validation. It is the standard way to measure how well a machine learning model will work on data it has never seen.

Cross-validation (CV) is a way to test a model many times on the same dataset. You split the data into parts. You train on some parts and test on the part the model did not see. Then you rotate, so every part gets a turn as the test set. Finally, you average the scores.

🌿
The Problem Cross-Validation Solves

With one 80/20 split, your accuracy depends on which rows ended up in the 20%. If the test rows happen to be easy, the score is too high. If they happen to be hard, the score is too low. Cross-validation averages over several test sets. This removes most of the luck and gives a much more reliable number.

Try It — The Lucky-Split Machine

Below is one small dataset (60 patients) and one simple model. The only thing that changes is which 12 rows are used for testing. Watch how much the accuracy jumps around with single splits, and how much calmer the 5-fold CV average is.

🎲 The Lucky-Split MachineSame data, same model — only the split changes.
● healthy   ● sick   ◯ = test row   — model boundary
Section 02

The Three Mistakes CV Protects You From

📈
Mistake 1 — Testing on Training Data
The overfitting trap
A model can memorise its training data, noise included. It may score 99% on training data and 61% on new data. If you only check training data, you report 99% and ship a bad model. CV always tests on rows the model did not train on.
🎲
Mistake 2 — Trusting One Lucky Split
The one-split gamble
One train/test split is one roll of the dice. Your 85% might be 80% or 90% with a different random seed. With a test set of 200 rows, the score can easily move by ±5 points by luck alone. CV averages several splits, so the number is steadier.
🔄
Mistake 3 — Wasting Data
The small-dataset problem
With 500 rows, a fixed 80/20 split tests on only 100 rows, forever. 5-fold CV also trains on 400 and tests on 100 — but it does this 5 times, so all 500 rows are used for testing once.
Section 03

Standard K-Fold Cross-Validation — The Foundation

K-Fold CV is the most common method. The data is cut into k equal parts called folds. The model trains on k−1 folds and is tested on the one fold left over. This repeats k times, so each fold is the test set exactly once. The final score is the mean of the k scores, and the standard deviation shows how much they vary.

01
Shuffle and Cut the Data into k Folds
N rows are cut into k parts that do not overlap. With k = 5 and 1,000 rows, each fold has 200 rows. Shuffle first, because data files are often sorted (by date, by class, by city).
02
Round i: Train on k−1 Folds, Test on Fold i
Fold i is the test set. A fresh model trains on all the other folds together. Its score on fold i (accuracy, AUC, RMSE…) is saved.
03
Rotate Through All k Folds
Repeat for i = 1, 2, …, k. Every row is in a test set exactly once, and in a training set k−1 times. No row is wasted.
04
Report Mean ± Standard Deviation
CV score = mean(score₁, …, scoreₖ). A large standard deviation means the model is sensitive to which rows it sees — a warning sign.
05
Train the Final Model on All the Data
CV is for measuring, not for building. The k fold-models are thrown away. Once you are happy with the score, train one model on 100% of the data and deploy it. The CV score is your honest estimate of how it will perform.

Try It — The K-Fold Rotator

Here are 20 rows. Each row of the grid on the left is one round of CV. Press Next fold to train and test one round at a time. Try k = 2, 5, 10 and 20.

🔄 K-Fold RotatorEvery row is tested exactly once.
k (number of folds)
■ train   ■ test   ● sick   ○ healthy

Python — Standard K-Fold (k = 5)

import numpy as np
from sklearn.datasets import make_classification
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import KFold, cross_val_score

# ── Dataset: 1000 samples, 20 features ──────────────
X, y = make_classification(
    n_samples=1000, n_features=20,
    n_informative=12, random_state=42
)

# ── Model ────────────────────────────────────────────
model = RandomForestClassifier(n_estimators=100, random_state=42)

# ── K-Fold: 5 folds, shuffled ────────────────────────
kf = KFold(n_splits=5, shuffle=True, random_state=42)

# ── cross_val_score trains and tests on every fold ───
scores = cross_val_score(model, X, y, cv=kf, scoring='accuracy')

print("Per-Fold Scores:")
for i, s in enumerate(scores, 1):
    print(f"  Fold {i}: {s:.4f}")

print(f"\nMean Accuracy : {scores.mean():.4f}")
print(f"Std Deviation : {scores.std():.4f}")
print(f"Lowest-Highest: {scores.min():.4f} - {scores.max():.4f}")
Output
Per-Fold Scores: Fold 1: 0.9100 Fold 2: 0.8900 Fold 3: 0.8200 Fold 4: 0.8750 Fold 5: 0.8700 Mean Accuracy : 0.8730 Std Deviation : 0.0299 Lowest-Highest: 0.8200 - 0.9100
💡
Read the Spread, Not Just the Mean

The same model scored from 82% to 91% across the five folds. A single 80/20 split could have shown you any of these numbers. Report the mean and the standard deviation: 87.3% ± 3.0.

🔢
Choosing k

k = 5 is the usual default. k = 10 trains on more data per fold (90% instead of 80%), so its estimate is a little less pessimistic, but it costs twice the training time. k = 3 is fine for very large datasets or very slow models. k = N (Leave-One-Out) trains N models — only practical for small data (Section 05). If unsure, start with k = 5.

Section 04

Stratified K-Fold — The Fix for Imbalanced Classes

📖 Story
The Unlucky Fold
A hospital dataset has 1,000 patients: 900 healthy and 100 with a rare disease. With 5 folds, each fold has 200 patients, so a fair fold should hold about 20 sick patients. Plain K-Fold shuffles at random. By bad luck, one fold might get only 10 sick patients and another 30. The fold with 10 gives a shaky recall score. If the file is sorted and you forget to shuffle, all 100 sick patients can end up in one fold — and four folds contain no sick patients at all.

Stratified K-Fold fixes this. It deals the sick patients out like playing cards, so every fold gets the same 10% as the full dataset — 20 sick patients each.

StratifiedKFold keeps the class percentages the same in every fold. This matters most when one class is rare. Plain K-Fold can produce folds with very few, or even zero, examples of the rare class.

Try It — Where Do the Sick Patients Go?

100 patients, 10 sick. The data file lists the sick patients last, as real files often do. Switch between the three methods, change k, and press Repeat 500× to see the worst case of each method.

⚖️ Stratified vs Plain K-Fold100 patients, only 10 sick. Where do the sick ones go?
k (number of folds)
■ healthy   ■ sick

Python — Stratified K-Fold

import numpy as np
from sklearn.datasets import make_classification
from sklearn.ensemble import GradientBoostingClassifier
from sklearn.model_selection import KFold, StratifiedKFold, cross_val_score

# ── Imbalanced dataset: about 90% class 0, 10% class 1 ──
X, y = make_classification(
    n_samples=2000, n_features=20, n_informative=10,
    weights=[0.9, 0.1],          # imbalanced!
    random_state=42
)
print(f"Class counts: {np.bincount(y)}")

# ── Compare: how many class-1 samples land in each test fold? ──
kf  = KFold(n_splits=5, shuffle=True, random_state=7)
skf = StratifiedKFold(n_splits=5, shuffle=True, random_state=7)
print("\nClass-1 samples per test fold:")
print("  KFold          :", [int(y[te].sum()) for _, te in kf.split(X, y)])
print("  StratifiedKFold:", [int(y[te].sum()) for _, te in skf.split(X, y)])

# ── Evaluate with F1 and AUC (better than accuracy for imbalance) ──
model = GradientBoostingClassifier(n_estimators=100, random_state=42)
f1_scores  = cross_val_score(model, X, y, cv=skf, scoring='f1')
auc_scores = cross_val_score(model, X, y, cv=skf, scoring='roc_auc')

print("\nStratified K-Fold (k=5) results:")
for i in range(5):
    print(f"  Fold {i+1}: F1={f1_scores[i]:.4f}  AUC={auc_scores[i]:.4f}")
print(f"\nMean F1  : {f1_scores.mean():.4f} ± {f1_scores.std():.4f}")
print(f"Mean AUC : {auc_scores.mean():.4f} ± {auc_scores.std():.4f}")
Output
Class counts: [1790 210] Class-1 samples per test fold: KFold : [40, 40, 44, 37, 49] StratifiedKFold: [42, 42, 42, 42, 42] Stratified K-Fold (k=5) results: Fold 1: F1=0.7164 AUC=0.9381 Fold 2: F1=0.5000 AUC=0.9104 Fold 3: F1=0.4918 AUC=0.9260 Fold 4: F1=0.6250 AUC=0.9594 Fold 5: F1=0.5085 AUC=0.8992 Mean F1 : 0.5683 ± 0.0886 Mean AUC : 0.9266 ± 0.0211
⚠️
Look at the Output Again

Plain KFold put between 37 and 49 rare-class rows in its test folds. StratifiedKFold put exactly 42 in every fold. Also notice the F1 score: it moves from 0.49 to 0.72 between folds even with stratification. With rare classes, scores are noisy — always report the spread.

🧩
What scikit-learn Does by Default

If you pass a number, like cv=5, to cross_val_score or cross_validate with a classifier, scikit-learn already uses StratifiedKFold — but without shuffling. For regression it uses plain KFold, also without shuffling. Passing StratifiedKFold(n_splits=5, shuffle=True, random_state=42) yourself makes the choice clear and repeatable.

Section 05

Leave-One-Out CV (LOOCV) — The Extreme Case

Leave-One-Out CV is K-Fold with k = N, the number of rows. Each round tests on one row and trains on all the others. This repeats N times, so you train N models. Each round's score is either 1 (correct) or 0 (wrong).

🔬 LOOCV on 5 Rows
Round 1
Train on [2,3,4,5] → test on [1] → score₁
Round 2
Train on [1,3,4,5] → test on [2] → score₂
Round 3
Train on [1,2,4,5] → test on [3] → score₃
Round 4
Train on [1,2,3,5] → test on [4] → score₄
Round 5
Train on [1,2,3,4] → test on [5] → score₅
Result
Final score = mean(score₁ … score₅)

Python — Leave-One-Out CV

from sklearn.model_selection import LeaveOneOut, cross_val_score
from sklearn.svm import SVC
from sklearn.datasets import load_iris

# ── Small dataset: LOOCV is practical here ────────────
X, y = load_iris(return_X_y=True)
print(f"Dataset size: {X.shape}")   # 150 samples → 150 models

model = SVC(kernel='rbf', C=1.0)
loo = LeaveOneOut()

# ── Each score is 1 (correct) or 0 (wrong) ────────────
scores = cross_val_score(model, X, y, cv=loo, scoring='accuracy')

print(f"Total iterations  : {len(scores)}")
print(f"Scores (first 10) : {scores[:10]}")
print(f"Mean Accuracy     : {scores.mean():.4f}")
print(f"Misclassified     : {int((scores == 0).sum())} / {len(scores)}")
Output
Dataset size: (150, 4) Total iterations : 150 Scores (first 10) : [1. 1. 1. 1. 1. 1. 1. 1. 1. 1.] Mean Accuracy : 0.9667 Misclassified : 5 / 150
PropertyK-Fold (k=5)K-Fold (k=10)LOOCV (k=N)
Training rows per model80% of data90% of dataall rows but one
Test rows per round20% of data10% of data1 row
Models trained510N
Pessimism (bias) of the estimateA littleLessLeast
Stability of the estimateGoodGoodCan be unstable
Compute costLowMediumVery high for big N
Best forGeneral useMedium datasetsVery small datasets
📌
LOOCV Is Not Automatically the Best

LOOCV trains on almost all the data, so it is the least pessimistic. But its N training sets are almost identical, so the estimate can still wobble a lot from one dataset to another. Research has found that 5- or 10-fold CV is usually as good or better — and much cheaper. Use LOOCV mainly when you have very few rows.

Section 06

Stratified Shuffle Split — Many Random Splits

StratifiedShuffleSplit makes a number of independent random train/test splits that you choose (for example 10 splits of 80/20), each keeping the class balance. Unlike K-Fold, the test sets can overlap: one row may be tested several times, and another row never. It is useful when the data is large and you want quick, repeated checks with a chosen test size.

🔀
Key Difference from K-Fold

K-Fold guarantees that every row is tested exactly once. StratifiedShuffleSplit does not. Each split is drawn fresh, so you control the number of splits and the test size separately — for example 20 splits with a 10% test set.

Python — StratifiedShuffleSplit

from sklearn.model_selection import StratifiedShuffleSplit, cross_val_score
from sklearn.datasets import make_classification
from sklearn.linear_model import LogisticRegression

X, y = make_classification(n_samples=5000, n_features=15,
                           weights=[0.8, 0.2], random_state=42)
model = LogisticRegression(max_iter=1000)

# ── 10 independent random stratified 80/20 splits ────
sss = StratifiedShuffleSplit(n_splits=10, test_size=0.2, random_state=42)
auc_scores = cross_val_score(model, X, y, cv=sss, scoring='roc_auc')

print("StratifiedShuffleSplit (10 runs x 20% test):")
for i, s in enumerate(auc_scores, 1):
    print(f"  Run {i:2d}: AUC = {s:.4f}")
print(f"\nMean AUC : {auc_scores.mean():.4f}")
print(f"Std      : {auc_scores.std():.4f}")

# ── How often was each sample in a test set? ─────────
import numpy as np
count = np.zeros(len(y), int)
for _, te in sss.split(X, y):
    count[te] += 1
print(f"\nNever tested: {(count == 0).sum()} samples | tested 2+ times: {(count >= 2).sum()} samples")
Output
StratifiedShuffleSplit (10 runs x 20% test): Run 1: AUC = 0.9367 Run 2: AUC = 0.9233 Run 3: AUC = 0.9349 Run 4: AUC = 0.9137 Run 5: AUC = 0.9393 Run 6: AUC = 0.9184 Run 7: AUC = 0.9206 Run 8: AUC = 0.9165 Run 9: AUC = 0.9297 Run 10: AUC = 0.9364 Mean AUC : 0.9269 Std : 0.0090 Never tested: 527 samples | tested 2+ times: 3144 samples

The last line shows the difference from K-Fold: after 10 random splits, 527 of the 5,000 rows were never tested, and more than 3,000 were tested two or more times.

Section 07

Group K-Fold — When Rows Are Not Independent

📖 Story (illustrative)
The Patient Who Appeared Twice
You train a model to detect pneumonia from chest X-rays. You have 2,000 X-rays from 400 patients — 5 scans per patient, from different visits. You run plain 5-fold CV and get an excellent score.

But look inside fold 1. Patient #47's Monday scan is in the training set, and their Friday scan is in the test set. The model has already seen this patient. It can recognise their body shape and simply copy the answer it learned. Of course it scores high — it is recognising people, not detecting disease.

On truly new patients, the score drops a lot. Group K-Fold keeps all scans of one patient in the same fold. If patient #47 is in the test fold, all five of their scans are there, and none are in training.

Try It — Group Leakage Lab

Each patient has several scans close together (joined by thin lines). The model is a simple "nearest neighbour": it copies the label of the most similar training scan. Compare KFold with GroupKFold, and with the truth on brand-new patients.

🩻 Group Leakage Lab40 patients, several scans each. Can the model cheat?
scans per patient
how different two scans of one patient are
● healthy   ● sick   lines join scans of one patient   ◯ test scan whose patient is also in training

Python — Group K-Fold

import numpy as np
from sklearn.model_selection import KFold, GroupKFold, cross_val_score
from sklearn.ensemble import RandomForestClassifier

# ── Simulate: 400 patients, 5 scans each ─────────────
rng = np.random.default_rng(42)
n_patients, scans = 400, 5
groups = np.repeat(np.arange(n_patients), scans)          # [0,0,0,0,0, 1,1,1,1,1, ...]

body   = rng.normal(size=(n_patients, 20))                # each patient's own "anatomy"
y_pat  = (body[:, 0] + rng.normal(scale=2.0, size=n_patients) > 0).astype(int)  # weak real signal
X = body[groups] + rng.normal(scale=0.3, size=(n_patients * scans, 20))         # 5 similar scans
y = y_pat[groups]

model = RandomForestClassifier(n_estimators=200, random_state=42)

# ── ❌ Plain KFold: scans of one patient land in train AND test ──
leaky = cross_val_score(model, X, y, cv=KFold(5, shuffle=True, random_state=42))
# ── ✅ GroupKFold: all scans of a patient stay in one fold ──
gkf = GroupKFold(n_splits=5)
honest = cross_val_score(model, X, y, cv=gkf, groups=groups)

print(f"KFold      (leaky) : {leaky.mean():.4f} ± {leaky.std():.4f}")
print(f"GroupKFold (honest): {honest.mean():.4f} ± {honest.std():.4f}")

# ── Verify: no patient appears in both train and test ─
for fold, (tr, te) in enumerate(gkf.split(X, y, groups), 1):
    leak = set(groups[tr]) & set(groups[te])
    print(f"  Fold {fold}: {len(set(groups[te]))} test patients, shared patients = {len(leak)}")
Output
KFold (leaky) : 0.9320 ± 0.0083 GroupKFold (honest): 0.5775 ± 0.0255 Fold 1: 80 test patients, shared patients = 0 Fold 2: 80 test patients, shared patients = 0 Fold 3: 80 test patients, shared patients = 0 Fold 4: 80 test patients, shared patients = 0 Fold 5: 80 test patients, shared patients = 0
⚠️
93% vs 58% — Same Model, Same Data

Plain KFold reported 93% because test scans had a "twin" from the same patient in training. GroupKFold reported the honest 58%. Use group-aware CV whenever one real-world unit appears in many rows: many scans per patient, many purchases per customer, many posts per user, many frames per video, many readings per machine. For classification with groups, StratifiedGroupKFold also keeps the class balance.

Section 08

Time-Series Cross-Validation — Respecting Time

📖 Story
The Forecaster Who Knew the Future
You build a model to forecast daily sales. You shuffle the data and run 5-fold CV. One fold puts January 2024 in training and July 2023 in the test set. The model is trained on the future and tested on the past.

In real life you always train on the past and predict the future. A model that peeks at future data looks brilliant in CV and then disappoints in production — because time only moves forward.

TimeSeriesSplit follows this rule: the training data is always older than the test data. No peeking.

Try It — Time-Series Split Explorer

Each row is one split over 24 months. Compare TimeSeriesSplit with shuffled KFold, and try the gap and sliding window options.

📅 Time-Series Split Explorer24 months of sales. Train on the past, test on the future.
n_splits
gap (months)
■ train   ■ test   ▢ gap   ! = training month that is AFTER the test month

Python — TimeSeriesSplit

import numpy as np
from sklearn.model_selection import TimeSeriesSplit, KFold, cross_val_score
from sklearn.ensemble import GradientBoostingRegressor
from sklearn.metrics import mean_absolute_error

# ── Simulate 360 days of sales: trend + monthly cycle + noise ──
rng = np.random.default_rng(42)
n = 360
t = np.arange(n)
X = np.column_stack([
    np.sin(2*np.pi*t/30),          # monthly cycle
    np.sin(2*np.pi*t/7),           # weekly cycle
    t/360,                         # time (trend)
    rng.normal(size=(n, 9))        # noise features
])
y = 10 + 0.02*t + 3*np.sin(2*np.pi*t/30) + rng.normal(scale=0.5, size=n)

model = GradientBoostingRegressor(n_estimators=100, random_state=42)
tscv = TimeSeriesSplit(n_splits=5)

maes = []
for fold, (tr, te) in enumerate(tscv.split(X), 1):
    model.fit(X[tr], y[tr])
    mae = mean_absolute_error(y[te], model.predict(X[te]))
    maes.append(mae)
    print(f"Fold {fold}: train days 0-{tr[-1]}  ->  test days {te[0]}-{te[-1]} | MAE={mae:.3f}")

print(f"\nTimeSeriesSplit mean MAE : {np.mean(maes):.3f}")

# ── ❌ Shuffled KFold lets the model see the future ───
kf = KFold(n_splits=5, shuffle=True, random_state=42)
mae_kf = -cross_val_score(model, X, y, cv=kf, scoring='neg_mean_absolute_error')
print(f"Shuffled KFold mean MAE  : {mae_kf.mean():.3f}   <- looks better, but it is cheating")
Output
Fold 1: train days 0-59 -> test days 60-119 | MAE=0.880 Fold 2: train days 0-119 -> test days 120-179 | MAE=0.962 Fold 3: train days 0-179 -> test days 180-239 | MAE=1.123 Fold 4: train days 0-239 -> test days 240-299 | MAE=0.819 Fold 5: train days 0-299 -> test days 300-359 | MAE=1.001 TimeSeriesSplit mean MAE : 0.957 Shuffled KFold mean MAE : 0.440 <- looks better, but it is cheating
📈
What This Output Teaches

Shuffled KFold reported an error of 0.44 — less than half the honest 0.96. It looked better only because it could see days around each test day, including future ones.

Also notice that the honest error did not fall as the training window grew. Sales keep rising, and tree models such as gradient boosting cannot predict values higher than anything they have seen. Much of each new test block is above the training range. TimeSeriesSplit exposes this problem; shuffled KFold hides it. If errors grow in later folds, also suspect drift — the data changing over time.

Section 09

Repeated K-Fold — For Steadier Estimates

RepeatedKFold and RepeatedStratifiedKFold run K-Fold several times, each time with a different shuffle, and average all the scores. 5 folds repeated 10 times gives 50 scores. The average is steadier because it no longer depends on one particular way of cutting the folds.

🔬
When to Use Repeated K-Fold

Use it when the dataset is small, or when you compare two models whose scores are close (for example 84.2% vs 84.7%). It costs n_repeats times more training, so on large datasets plain 5-fold is usually enough.

Python — Comparing Two Models Fairly

import numpy as np
from scipy import stats
from sklearn.model_selection import RepeatedStratifiedKFold, cross_val_score
from sklearn.ensemble import RandomForestClassifier
from sklearn.neighbors import KNeighborsClassifier
from sklearn.datasets import make_classification

X, y = make_classification(n_samples=500, n_features=20,
                           n_informative=10, random_state=42)

# ── 5 folds x 10 repeats = 50 scores per model ────────
k, r = 5, 10
rskf = RepeatedStratifiedKFold(n_splits=k, n_repeats=r, random_state=42)

rf  = cross_val_score(RandomForestClassifier(n_estimators=100, random_state=42), X, y, cv=rskf)
knn = cross_val_score(KNeighborsClassifier(n_neighbors=5), X, y, cv=rskf)
print(f"Random Forest : {rf.mean():.4f} ± {rf.std():.4f}")
print(f"kNN (k=5)     : {knn.mean():.4f} ± {knn.std():.4f}")

# ── The 50 scores share training data, so they are NOT independent.
# ── Use the corrected resampled t-test (Nadeau & Bengio, 2003).
d = knn - rf
n_test = len(y) // k
n_train = len(y) - n_test
var = d.var(ddof=1)
t_naive = d.mean() / np.sqrt(var / len(d))
t_corr  = d.mean() / np.sqrt((1/len(d) + n_test/n_train) * var)
p_naive = 2 * stats.t.sf(abs(t_naive), df=len(d) - 1)
p_corr  = 2 * stats.t.sf(abs(t_corr),  df=len(d) - 1)
print(f"\nMean difference : {d.mean():+.4f}")
print(f"Naive t-test    : p = {p_naive:.4f}   (too optimistic)")
print(f"Corrected t-test: p = {p_corr:.4f}")
Output
Random Forest : 0.8812 ± 0.0315 kNN (k=5) : 0.9026 ± 0.0280 Mean difference : +0.0214 Naive t-test : p = 0.0000 (too optimistic) Corrected t-test: p = 0.1766
⚠️
Do Not Use a Plain t-Test on CV Scores

The 50 scores are not independent: every training set shares most of its rows with the others. A normal paired t-test ignores this and becomes far too confident. Here it says p ≈ 0.0000, "kNN is clearly better". The corrected resampled t-test (Nadeau & Bengio, 2003) adds the term n_test / n_train to the variance. It says p = 0.18: the 2-point difference could easily be luck.

Section 10

Nested Cross-Validation — Honest Scores After Tuning

📖 Story
The Teacher Who Marks Their Own Exam
You use 5-fold CV to tune your model. You try 100 hyperparameter settings and keep the one with the best CV score. Then you report that best score as your model's performance.

There is a problem. You chose that setting because it did well on these exact folds. Part of its success is luck. So the best score is usually higher than what you will get on truly new data. This is called optimistic selection bias.

Nested CV uses two loops. The inner loop tunes the hyperparameters. The outer loop measures performance on folds that the tuning never touched.
📊 Nested CV — The Outer Loop Measures, the Inner Loop Tunes
OUTER LOOP — measures performance (5 folds) TEST outer TRAIN (folds 2–5) ← outer fold 1 ⋮ ← outer folds 2–5: same idea INNER LOOP — tunes hyperparameters (3 folds, inside outer TRAIN only) validation inner train validation inner train validation each setting is scored on 3 inner folds ① Inner loop picks the best setting → ② retrain it on all outer TRAIN rows → ③ score it once on the outer TEST fold Final: average of the 5 outer TEST scores = honest estimate for "my tuning procedure + model"

5 outer folds × 3 inner folds = 15 model fits for every hyperparameter setting. With 18 settings that is 270 fits, plus 5 refits. Nested CV is slow, but it is the honest way to report a score when tuning and evaluation happen together.

Try It — The Optimism Gap

This dataset is pure noise: the labels are coin flips, so the true accuracy of any model is 50%. Each "setting" is a model that uses 3 random features. Try more and more settings, keep the best CV score, and compare it with nested CV.

🎯 The Optimism Gap (why nested CV)Pure noise data. Try many settings and keep the best CV score. Averaged over 20 datasets.
settings tried
rows in the dataset

Python — Nested CV with GridSearchCV

from sklearn.model_selection import StratifiedKFold, GridSearchCV, cross_val_score
from sklearn.ensemble import RandomForestClassifier
from sklearn.datasets import make_classification

X, y = make_classification(n_samples=600, n_features=20,
                           n_informative=10, random_state=42)

outer_cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)   # honest evaluation
inner_cv = StratifiedKFold(n_splits=3, shuffle=True, random_state=1)    # tuning

param_grid = {
    'n_estimators': [50, 100, 200],
    'max_depth':    [3, 5, None],
    'max_features': ['sqrt', 'log2']
}   # 3x3x2 = 18 combinations x 3 inner folds = 54 fits per outer fold

# ── GridSearchCV is the inner loop ────────────────────
clf = GridSearchCV(RandomForestClassifier(random_state=42),
                   param_grid, cv=inner_cv, scoring='accuracy', n_jobs=-1)

# ── cross_val_score is the outer loop ─────────────────
nested = cross_val_score(clf, X, y, cv=outer_cv, scoring='accuracy')

print("Nested CV outer scores:")
for i, s in enumerate(nested, 1):
    print(f"  Outer fold {i}: {s:.4f}")
print(f"\nNested CV mean : {nested.mean():.4f} ± {nested.std():.4f}")

# ── Non-nested: tune on all data, report the best inner score ──
clf.fit(X, y)
print(f"\nNon-nested best CV score : {clf.best_score_:.4f}   <- picked as the best, so it is optimistic")
print(f"Nested CV estimate       : {nested.mean():.4f}")
print(f"Difference               : {clf.best_score_ - nested.mean():+.4f}")
Output
Nested CV outer scores: Outer fold 1: 0.9583 Outer fold 2: 0.9000 Outer fold 3: 0.9000 Outer fold 4: 0.8833 Outer fold 5: 0.9000 Nested CV mean : 0.9083 ± 0.0258 Non-nested best CV score : 0.8917 <- picked as the best, so it is optimistic Nested CV estimate : 0.9083 Difference : -0.0167
🎯
A Surprise in the Output

Here the non-nested score (0.892) is actually lower than the nested score (0.908). So the gap is not always positive on one dataset — both numbers have their own noise. The bias shows up on average, and it grows with the number of settings you try, as the simulator above shows. With a small grid like this one it is small. With a large search it can be several points. When you tune and report together, nested CV (or a separate untouched test set) is the honest choice.

Section 11

cross_validate — Many Metrics in One Call

cross_val_score returns one metric. cross_validate can return several metrics at once, plus the training scores and the time each fold took. Use it whenever you need more than one number per fold.

Python — cross_validate with Several Metrics

from sklearn.model_selection import StratifiedKFold, cross_validate
from sklearn.ensemble import GradientBoostingClassifier
from sklearn.datasets import make_classification

X, y = make_classification(n_samples=2000, n_features=20,
                           weights=[0.8, 0.2], random_state=42)

model = GradientBoostingClassifier(n_estimators=100, random_state=42)
skf   = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)

# ── Five metrics in one call, plus train scores ──────
metrics = ['accuracy', 'f1', 'roc_auc', 'precision', 'recall']
res = cross_validate(model, X, y, cv=skf, scoring=metrics,
                     return_train_score=True)   # train scores help spot overfitting

print(f"{'Metric':10s}  {'Train':>6s}  {'Test':>6s}  {'Gap':>7s}")
print("-" * 36)
for m in metrics:
    tr, te = res[f'train_{m}'].mean(), res[f'test_{m}'].mean()
    print(f"{m:10s}  {tr:.4f}  {te:.4f}  {tr-te:+.4f}")
Output
Metric Train Test Gap ------------------------------------ accuracy 0.9784 0.9310 +0.0474 f1 0.9459 0.8266 +0.1193 roc_auc 0.9972 0.9620 +0.0352 precision 0.9558 0.8449 +0.1109 recall 0.9363 0.8144 +0.1219
🔍
Reading the Train–Test Gap

The gap between train and test scores is your overfitting signal. Accuracy has a small gap (0.05), but F1, precision and recall have gaps of about 0.12. The model is memorising part of the rare class. More regularisation, a simpler model or more data would help. With return_train_score=True, this check costs nothing extra.

Section 12

CV with Pipelines — The Only Safe Way

⚠️
The Most Common CV Mistake — Leakage Through Preprocessing

Any step that learns from the data — scaling, filling missing values, choosing features, target encoding — must be learned on the training fold only. If you run it on all rows before CV, the test rows have already influenced the model. The score becomes too optimistic. Put every such step inside a Pipeline, so CV refits it on each training fold.

Try It — Preprocessing Leakage

60 rows of pure noise, so the honest accuracy is 50%. Compare "prepare on all rows, then CV" with "prepare inside each fold". Increase the number of features, and switch between feature selection and scaling.

🚰 Preprocessing Leakage60 rows of pure noise (average of 8 datasets). The honest answer is 50%.
number of noise features

Python — Wrong vs Right

import numpy as np
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.feature_selection import SelectKBest, f_classif
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import StratifiedKFold, cross_val_score

# ── Pure noise: 100 samples, 5000 random features, random labels ──
# ── No model can truly beat 50% here.
rng = np.random.default_rng(0)
X = rng.normal(size=(100, 5000))
y = rng.integers(0, 2, size=100)
skf = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)

# ─── ❌ WRONG: pick the 20 "best" features using ALL rows, then CV ──
X_sel = SelectKBest(f_classif, k=20).fit_transform(X, y)   # test rows leak here!
wrong = cross_val_score(LogisticRegression(max_iter=1000), X_sel, y, cv=skf)

# ─── ✅ RIGHT: selection + scaling happen inside each training fold ──
pipe = Pipeline([
    ('scale',  StandardScaler()),
    ('select', SelectKBest(f_classif, k=20)),
    ('model',  LogisticRegression(max_iter=1000)),
])
right = cross_val_score(pipe, X, y, cv=skf)

print(f"❌ Select features first, then CV : {wrong.mean():.2f}")
print(f"✅ Pipeline inside CV              : {right.mean():.2f}")
print("   (the true accuracy on noise is 0.50)")
Output
❌ Select features first, then CV : 0.86 ✅ Pipeline inside CV : 0.44 (the true accuracy on noise is 0.50)
✅
How Big Is the Leak?

It depends on the step. Choosing features on all rows turned pure noise into "86% accuracy" — a huge lie. Scaling on all rows usually leaks very little. You cannot easily tell in advance, so make the Pipeline a habit for every step that learns from data.

Section 13

All CV Methods at a Glance — Decision Guide

Answer the five questions below. The guide picks a splitter and writes the scikit-learn code for you.

🧭 Which Cross-Validation Should I Use?Answer five questions. Get a splitter and the code.
Is it a classification problem (predicting a class)?
Can one person, customer or device appear in many rows?
Does the order in time matter (forecasting)?
Is the dataset small (a few hundred rows or less)?
Will you tune hyperparameters and also report the score?
CV MethodKeeps Class BalanceGroup-SafeTime-SafeCompute CostBest For
KFoldNoNoNoLowRegression
StratifiedKFoldYesNoNoLowClassification (default choice)
LeaveOneOutNoNoNoVery highVery small datasets
StratifiedShuffleSplitYesNoNoYou chooseLarge datasets, quick repeated checks
GroupKFoldNoYesNoLowPatient / user / device data
StratifiedGroupKFoldYesYesNoLowGrouped classification data
TimeSeriesSplitNoNoYesLowTime-ordered data, forecasting
RepeatedStratifiedKFoldYesNoNoHighSmall data, close model comparisons
Nested CVYesIf the inner/outer splitters areIf the inner/outer splitters areVery highTuning + honest reporting
Section 14

Golden Rules of Cross-Validation

🌿 Cross-Validation — Rules You Must Know
1
For classification, use StratifiedKFold with shuffling. It keeps the class balance in every fold. Write it out explicitly, e.g. StratifiedKFold(5, shuffle=True, random_state=42), so your results are clear and repeatable.
2
Put every preprocessing step inside a Pipeline. Scaling, filling missing values, encoding, feature selection, PCA — anything that learns from data. Outside a Pipeline it leaks test information and inflates the score.
3
CV is for measuring, not for deploying. After CV tells you the model is good, train one final model on 100% of the data. Do not ship the fold models — each one saw only (k−1)/k of the data.
4
Look at the standard deviation, not just the mean. 85% ± 1 is a stable model. 85% ± 8 means the result swings a lot with the data — expect surprises in production.
5
Find the grouping and time structure before choosing a splitter. Many rows per patient, user or device → GroupKFold. Time-ordered data → TimeSeriesSplit. Plain CV on such data is one of the most common and most damaging mistakes.
6
k = 5 is a good default. Use k = 10 when you can afford twice the compute. Use Leave-One-Out only for very small data. Avoid k = 2: each model trains on only half the data, so the score is pessimistic and noisy.
7
Do not report the best tuning score as your final score. Picking the best of many settings and reporting that same score is double-dipping. Use nested CV, or keep a separate test set that the tuning never touches.
8
Touch the final test set only once, at the very end. CV is your development tool. If you test, change the model, and test again, the test set has become part of training and is no longer honest.