Data Preparation / Data Preprocessing Slides 📂 Introduction · 13 of 13 44 min read

Data Splitting Mastery — Train, Validation & Test

A practical, visual guide to evaluating models honestly. Master train/validation/test splits, stratification, K-fold and Stratified K-fold, LOOCV, time-series splits and nested CV — and shut down data leakage with a split-first, Pipeline-driven workflow. Pick the right technique for any dataset size or task.

Data Splitting Mastery

A model is only as trustworthy as the way you tested it. Train, validate and test the honest way — holdouts, cross-validation, time-series splits — and never let the test set leak.
Train / Val / Test K-Fold & Stratified Time-Series Leakage

Press Next → or use ← → arrow keys

Section 01

Why Data Splitting Matters

🩺 REAL STORY · 98% → 61%
The doctor who tested on his own patients
Dr. Aryan trained a cancer detector on 1,000 records and tested on those same records — 98% accuracy. In deployment it fell to 61%. The model had memorised patient IDs and data-entry quirks, not cancer. A proper split would have exposed it before any harm.
🔒
The golden rule

Never use the test set for any development decision — not hyperparameters, not feature selection, not architecture. Peek even once and the test set becomes a second validation set and your reported score becomes fiction.

📝
The exam analogy

Studying the exact exam paper then sitting it scores 100% — but proves nothing. A fair exam uses new questions. Your test set is that new exam.

Section 01 · Framework

Train · Validation · Test

Training · 60%weights & boundaries fitted here Val · 20%tune here Test · 20%touched once 🔒
🏋️
Training
The model learns here — weights, coefficients and decision boundaries are all fitted on this set.
🎛️
Validation
Tune hyperparameters and compare architectures. Observed repeatedly — a feedback loop, not a judge.
🏆
Test
Locked away until the end. Touched exactly once for the final, honest generalisation number.
📐
Proportions are guidelines

1M+ rows tolerate a 98/1/1 split; under 1,000 rows, prefer cross-validation over a static holdout.

Section 02

Train-Test Split — the Holdout Method

A spam filter hit 99.2% training on 5,000 emails — then caught only 60% of real spam, because sender addresses it had memorised never reappeared. Shuffle, then split, and you'd see it instantly.

1 · raw (ordered) S1 S2 S3 … 2 · shuffle (seed 42) S7 S2 S9 … 80% train 20% Always shuffle before splitting — except time-series, where order must be preserved 
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.20, random_state=42, shuffle=True, stratify=y)
Section 02 · Stratify

Stratified Split — Keep the Class Balance

With a 5% fraud class, a random split can — by pure chance — put zero frauds in the test set. stratify=y guarantees both sides mirror the original ratio exactly.

Without stratify · risky
SetClass 0Fraud
Train76040 (5%)
Test19010 — or 0!
Worst8000 (0%)
With stratify=y · safe
SetClass 0Fraud
Train76040 (5%)
Test19010 (5%)
Bothexact same ratio ✓
⚖️
Non-negotiable for classification

Any time the target is categorical — especially when it's imbalanced — pass stratify=y. A test set with no minority cases can't measure minority-class performance at all.

Section 02 · Ratios

How Much to Hold Out?

RULE OF THUMB BY DATASET SIZE
< 1Kuse cross-validation — you can't afford a static holdout
1K–10K70 / 30 split
10K–100K80 / 20 split
> 100K90 / 10 or even 99 / 1 — plenty of test rows remain
🎚️
The trade-off

More training data helps the model learn, but leaves fewer rows to evaluate on. On big data the test fraction can shrink because even 1% is thousands of examples; on small data every row is precious, so cross-validation reuses them all.

# key parameters
train_test_split(X, y, test_size=0.2, random_state=42, shuffle=True, stratify=y)
# test_size · random_state (reproducible!) · shuffle (False for time-series) · stratify
Section 03

The Validation Set — Tune Without Cheating

A scientist tested 50 hyperparameter combos, each time scoring on the test set and keeping the best. Test accuracy looked superb; production was terrible. Fifty peeks = she effectively trained on the test set.

Model A · lr=.01 Model B · lr=.1 Model C · d=8 Validation setcompare · pick winner Test set — oncefinal honest number Every model is judged on validation; only the winner ever touches the test set.
📉
Test slightly below validation is healthy

Validation selected the winner, so it's a touch optimistic; the test set is the honest estimate. If test beats validation, suspect a bug or a leak.

Section 04

K-Fold Cross-Validation

5-fold — the validation fold rotates through every slice once F1 F2 F3 F4 F5 validationtrain 
kf = KFold(n_splits=5, shuffle=True, random_state=42)
scores = cross_val_score(model, X, y, cv=kf)   # [.88 .91 .89 .90 .88]
print(f"{scores.mean():.3f} ± {2*scores.std():.3f}")   # mean ± 95% CI · high σ = unstable
Section 05

Stratified K-Fold — Balanced Folds

Regular K-Fold — ratios drift ✗ F1: fraud present F2: 0 fraud — disaster! F3: fraud bunched Stratified — same ratio each fold ✓ every fold ≈ 80% majority · 20% minority majorityminority 
skf = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)
cross_val_score(model, X, y, cv=skf, scoring='roc_auc')   # every fold sees the right ratio
✅
The classification default

Almost always prefer StratifiedKFold over plain KFold for classification — plain K-Fold can hand a fold zero positive examples and quietly ruin the score. (For regression, stratifying a continuous target isn't meaningful — use plain KFold.)

Section 06

Leave-One-Out CV — for Tiny Datasets

Dr. Singh has just 10 patient records. An 80/20 split leaves 2 to validate on — useless. LOOCV trains on 9, tests on the 10th, and repeats 10 times so every patient serves as the test case once.

n = 6 · the single test sample rotates each iteration i=1 i=2 i=3 i=N test (1 sample)train (n−1)
⏳
Least biased, but expensive

LOOCV uses the maximum training data every time — ideal under ~100 rows. But it runs N times, so on large data it's prohibitive: prefer 5- or 10-fold there.

Section 07

Time-Series Split — Never Peek Ahead

Vivek's stock model hit 94% with random K-Fold — then lost money. It had trained on future prices to predict the past. TimeSeriesSplit revealed the real accuracy: 52%, barely better than a coin flip.

Expanding window — train on the past, validate on the future S1 S2 S3 S4 Past 2020Future 2024 → train (past)validate (future) 
tscv = TimeSeriesSplit(n_splits=5)   # never shuffle time-series!
cross_val_score(Ridge(), X, y, cv=tscv, scoring='neg_root_mean_squared_error')
Section 08

Which Technique Should You Use?

Time-series data? → TimeSeriesSplit tiny (<100 rows)? → LOOCV classification? → Stratified K-Fold huge / deep learning? yes → Train/Val/Test · else → K-Fold
🧭
When in doubt: 5- or 10-fold CV

Plain (or stratified) k-fold is robust, widely understood, and the right call for roughly 90% of projects.

Section 09

Nested CV — Tune & Estimate Honestly

When one CV loop both tunes hyperparameters and reports performance, the score is optimistically biased. Nested CV separates the two: an outer loop measures, an inner loop searches.

OUTER · unbiased estimate test fold inner training pool (folds 2–5) INNER · GridSearchCV val inner train → best params → score outer test 
clf = GridSearchCV(SVC(), param_grid, cv=inner_cv)          # inner: tune
cross_val_score(clf, X, y, cv=outer_cv)                    # outer: estimate → 0.89 ± 0.01
Section 10

Data Leakage — the #1 Mistake in ML

🎯
Target Leakage
A feature secretly derived from the outcome — e.g. using claim_amount to predict will_claim. Most dangerous.
🔀
Train-Test Contamination
Fitting a scaler / encoder / imputer on the full data before splitting. Test stats bleed into training.
📅
Temporal Leakage
Future information used to predict the past — usually from shuffling sequential data.
🕵️
The leaky-exam analogy

A student who saw next week's paper aces it — then fails the real exam. Leakage lets your model "cheat" on information it won't have in production, so brilliant dev scores collapse in the real world.

Section 10 · Fix

Split First, Then Fit — Every Time

❌ Leaky — fit on all, then split
scaler.fit(X_all)
X = scaler.transform(X_all)
split(X) → train / test
test stats already leaked
✅ Clean — split, then fit on train
split(X_raw) → train / test
scaler.fit(X_train)
transform train & test
test never seen at fit
# a Pipeline enforces it automatically —
# the scaler is re-fit on TRAIN inside each fold
pipe = Pipeline([
  ('scaler', StandardScaler()),
  ('model', GradientBoostingClassifier())])

cross_val_score(pipe, X, y,
  cv=StratifiedKFold(5, shuffle=True))  # leak-free
🧷
Pipelines are the seatbelt

Inside CV, a Pipeline re-fits every transformer on each fold's training data only — killing the most common leak by design.

Section 11 · Part 1

The Golden Rules — 1 to 3

✂️ DATA SPLITTING · RULES 1–3
1
Split before any preprocessing. Fit scalers, imputers and encoders on training data only, then apply to val/test without refitting.
2
Set random_state everywhere. A split without a seed is a split you can't reproduce — and reproducibility isn't optional.
3
Use stratify for classification. Categorical target → stratify=y in splits and StratifiedKFold in CV.
Section 11 · Part 2

The Golden Rules — 4 to 6 & Takeaway

✂️ DATA SPLITTING · RULES 4–6
4
Never tune on the test set. Touch it more than once and it becomes a second validation set — your reported score turns optimistic.
5
Respect temporal order. Past trains, future validates — never shuffle time-series before splitting.
6
Use sklearn Pipelines. They apply every step correctly inside each fold and eliminate the most common source of leakage.
🎯
Splitting is a philosophy of honesty

The gold-standard workflow: split off the test set first → Stratified K-Fold + Pipeline on the training data → pick the best model → evaluate once on test → report. A model evaluated properly is a model you can actually trust.

✂️ End of tutorial · Press ← to review, or click Restart

1 / 17
You have completed Introduction. View all sections →