Data Splitting Mastery

Why Data Splitting Matters ✂️Data Splitting क्यों ज़रूरी है ✂️为什么要划分数据 ✂️

A model is useful only if it works on data it has never seen. If you score it on the rows it learned from, you measure memory, not skill. Data splitting keeps some rows aside so that training, tuning and the final check each use different data.

Model तभी काम का है जब वह अनदेखे data पर चले। अगर आप उसे उन्हीं rows पर परखते हैं जिनसे उसने सीखा, तो आप याददाश्त मापते हैं, हुनर नहीं। Data splitting कुछ rows अलग रखता है ताकि training, tuning और अंतिम जाँच हर एक अलग data इस्तेमाल करे।

模型只有在没见过的数据上有效才有用。如果用它学过的行来打分，测的是记忆而不是能力。数据划分把一部分行留在一边，让训练、调参和最终检验各自使用不同的数据。

# 1) Testing on the training data: memorising looks like learning
from sklearn.model_selection import train_test_split
from sklearn.tree import DecisionTreeClassifier
from sklearn.neighbors import KNeighborsClassifier
from sklearn.metrics import accuracy_score
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42, stratify=y)
for name, m in [("decision tree (no depth limit)", DecisionTreeClassifier(random_state=0)),
                ("1-nearest neighbour", KNeighborsClassifier(1))]:
    m.fit(X_train, y_train)
    print(f"{name:<31} accuracy on its training rows {accuracy_score(y_train, m.predict(X_train)):.3f} | on unseen test rows {accuracy_score(y_test, m.predict(X_test)):.3f}")

Train-Test Split (Holdout) 🔀Train-Test Split (Holdout) 🔀训练-测试划分（留出法）🔀

The simplest method: shuffle the rows once and keep a share aside for testing. Set random_state so the split is reproducible, and for classification add stratify=y so both parts keep the same class mix. Do not shuffle time-ordered data (Section 7).

सबसे सरल तरीका: rows को एक बार shuffle करें और एक हिस्सा testing के लिए अलग रखें। random_state तय करें ताकि split दोहराया जा सके, और classification में stratify=y जोड़ें ताकि दोनों हिस्सों में class का मिश्रण एक जैसा रहे। समय-क्रम वाले data को shuffle न करें (Section 7)।

最简单的方法：把行打乱一次，留出一部分用于测试。设置 random_state 以便复现划分；分类问题加上 stratify=y，让两部分的类别比例相同。不要打乱有时间顺序的数据（第 7 节）。

# 2) train_test_split, and what stratify=y really guarantees
print(f"train {len(X_train)} rows, test {len(X_test)} rows | malignant share: train {y_train.mean():.3f}, test {y_test.mean():.3f}")
plain = [train_test_split(X, y, test_size=0.2, random_state=s)[3].mean() for s in range(1000)]
strat = [train_test_split(X, y, test_size=0.2, random_state=s, stratify=y)[3].mean() for s in range(1000)]
print(f"malignant share in the test set over 1,000 random splits (full data {y.mean():.3f}):")
print(f"  without stratify: {min(plain):.3f} to {max(plain):.3f}")
print(f"  with stratify=y : {min(strat):.3f} to {max(strat):.3f}")

One split is a lottery ticketएक split एक lottery ticket है一次划分就是一张彩票

# 3) One holdout split is a lottery ticket — the same model, 200 different test sets
from sklearn.linear_model import LogisticRegression
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import make_pipeline
model = make_pipeline(StandardScaler(), LogisticRegression(max_iter=5000))
accs = []
for s in range(200):
    a, b, c, d = train_test_split(X, y, test_size=0.2, random_state=s, stratify=y)
    accs.append(accuracy_score(d, model.fit(a, c).predict(b)))
accs = np.array(accs)
print(f"test accuracy over 200 splits: min {accs.min():.3f}, median {np.median(accs):.3f}, max {accs.max():.3f}")
print(f"with 114 test rows, one patient = {1 / 114:.3%} of accuracy")

The Validation Set 🎛️Validation Set 🎛️验证集 🎛️

As soon as you tune anything — tree depth, regularisation, which features — you need data to compare the options that is neither the training data nor the test set. Split off the test set first, then split the rest into training and validation.

जैसे ही आप कुछ भी tune करते हैं — tree depth, regularisation, कौन से features — आपको विकल्पों की तुलना के लिए ऐसा data चाहिए जो न training data हो न test set। पहले test set अलग करें, फिर बाक़ी को training और validation में बाँटें।

一旦要调任何东西 —— 树深度、正则化、用哪些特征 —— 就需要既不是训练集也不是测试集的数据来比较选项。先切出测试集，再把剩下的分成训练集和验证集。

# 1) test set first (20%), 2) then 25% of the rest = 20% of all for validation → 60 / 20 / 20
X_temp, X_test, y_temp, y_test = train_test_split(X, y, test_size=0.20, random_state=42, stratify=y)
X_train, X_val, y_train, y_val = train_test_split(X_temp, y_temp, test_size=0.25, random_state=42, stratify=y_temp)

K-Fold Cross-Validation 🔁K-Fold Cross-Validation 🔁K 折交叉验证 🔁

Split the data into k equal folds. Train on k − 1 of them, validate on the one left out, and repeat until every fold has been the validation fold once. Report the mean of the k scores. Every row is used for validation exactly once and for training k − 1 times.

Data को k बराबर folds में बाँटें। उनमें से k − 1 पर train करें, बचे एक पर validate करें, और तब तक दोहराएँ जब तक हर fold एक बार validation fold न बन जाए। k scores का mean बताएँ। हर row ठीक एक बार validation के लिए और k − 1 बार training के लिए इस्तेमाल होती है।

把数据分成 k 个相等的折。用其中 k − 1 折训练，留下的一折验证，重复直到每一折都当过一次验证折。报告 k 个分数的平均值。每一行恰好验证一次、训练 k − 1 次。

# 4) K-fold cross-validation: every row is validated exactly once
from sklearn.model_selection import KFold, StratifiedKFold, RepeatedStratifiedKFold, cross_val_score
skf = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)
scores = cross_val_score(model, X, y, cv=skf, scoring="accuracy")
print("fold scores :", np.round(scores, 4))
print(f"mean {scores.mean():.4f} | std across folds {scores.std():.4f}")
rep = cross_val_score(model, X, y, cv=RepeatedStratifiedKFold(n_splits=5, n_repeats=20, random_state=0), scoring="accuracy")
print(f"repeated 5-fold (×20): mean {rep.mean():.4f}, the 20 repeat-means range {rep.reshape(20, 5).mean(1).min():.4f}–{rep.reshape(20, 5).mean(1).max():.4f}")

Stratified K-Fold ⚖️Stratified K-Fold ⚖️分层 K 折 ⚖️

Plain KFold cuts the rows in file order unless you set shuffle=True. If the file is sorted by label — common in exports — folds can contain only one class. StratifiedKFold shuffles (when asked) and gives every fold the same class share. It is the default for classification in cross_val_score.

सादा KFold rows को file के क्रम में काटता है, जब तक आप shuffle=True न दें। अगर file label से sorted है — exports में आम — तो folds में सिर्फ़ एक class हो सकती है। StratifiedKFold (कहने पर) shuffle करता है और हर fold को एक जैसा class हिस्सा देता है। cross_val_score में classification के लिए यही default है।

普通的 KFold 在不设 shuffle=True 时按文件顺序切分。如果文件按标签排序 —— 导出数据中很常见 —— 某些折可能只含一个类别。StratifiedKFold（在要求时）会打乱，并且让每折的类别比例相同。它是 cross_val_score 对分类问题的默认设置。

from sklearn.metrics import roc_auc_score
# 5) When the rows are sorted by label: KFold without shuffle vs shuffle vs StratifiedKFold
order = np.argsort(y.to_numpy(), kind="stable")                  # all benign first, then all malignant
Xs, ys = X.iloc[order], y.iloc[order]
for name, cv in [("KFold(shuffle=False)", KFold(5)), ("KFold(shuffle=True)", KFold(5, shuffle=True, random_state=0)),
                 ("StratifiedKFold", StratifiedKFold(5, shuffle=True, random_state=0))]:
    share = [ys.iloc[v].mean() for _, v in cv.split(Xs, ys)]
    aucs = []
    for tr, va in cv.split(Xs, ys):
        p = model.fit(Xs.iloc[tr], ys.iloc[tr]).predict_proba(Xs.iloc[va])[:, 1]
        aucs.append(f"{roc_auc_score(ys.iloc[va], p):.2f}" if ys.iloc[va].nunique() == 2 else "n/a")
    print(f"{name:<22} malignant share per fold {np.round(share, 2)} | ROC-AUC per fold {aucs}")

Leave-One-Out (LOOCV) 1️⃣Leave-One-Out (LOOCV) 1️⃣留一法（LOOCV）1️⃣

LOOCV is k-fold with k = N: train on all rows but one, predict that one, repeat N times. It uses the most training data per fit, so its estimate has very little bias. But each "fold" score is just right or wrong (1 or 0), the N models are nearly identical, and you cannot compute metrics like ROC-AUC per fold. Its estimate can have high variance, and it costs N fits.

LOOCV, k = N वाला k-fold है: एक को छोड़ सभी rows पर train करें, उस एक को predict करें, N बार दोहराएँ। यह हर fit में सबसे ज़्यादा training data इस्तेमाल करता है, इसलिए इसके अनुमान में bias बहुत कम है। पर हर "fold" score बस सही या ग़लत है (1 या 0), N models लगभग एक जैसे हैं, और हर fold पर ROC-AUC जैसे metrics नहीं निकल सकते। इसके अनुमान का variance ऊँचा हो सकता है, और यह N fits लेता है।

LOOCV 就是 k = N 的 K 折：每次留下一行，用其余所有行训练、预测这一行，重复 N 次。每次拟合用的训练数据最多，所以估计的偏差很小。但每一“折”的分数只有对或错（1 或 0），N 个模型几乎相同，也无法按折计算 ROC-AUC 之类的指标。其估计的方差可能很高，而且需要 N 次拟合。

# 6) Leave-one-out on a tiny sample of 40 patients
from sklearn.model_selection import LeaveOneOut, cross_val_predict
import time
r = np.random.default_rng(3)
small = r.choice(len(X), 40, replace=False)
Xm, ym = X.iloc[small], y.iloc[small]
t0 = time.perf_counter(); loo = cross_val_score(model, Xm, ym, cv=LeaveOneOut()); t1 = time.perf_counter()
print(f"LOOCV: {len(loo)} fits, each 'fold' score is 0 or 1 → mean {loo.mean():.3f} ({t1 - t0:.2f} s)")
ten = [cross_val_score(model, Xm, ym, cv=StratifiedKFold(10, shuffle=True, random_state=s)).mean() for s in range(20)]
print(f"10-fold, 20 different shuffles: {min(ten):.3f} to {max(ten):.3f}")

Time Series Cross-Validation 📅Time Series Cross-Validation 📅时间序列交叉验证 📅

When rows are ordered in time, today is related to yesterday and tomorrow. Shuffled k-fold puts validation days between training days, so the model effectively sees the neighbours of each day it is tested on. TimeSeriesSplit always trains on the past and validates on the block that follows.

जब rows समय के क्रम में हों, तो आज कल और परसों से जुड़ा होता है। Shuffled k-fold validation days को training days के बीच रखता है, इसलिए model असल में हर test day के पड़ोसी देख लेता है। TimeSeriesSplit हमेशा अतीत पर train करता है और उसके बाद वाले block पर validate करता है।

当行按时间排序时，今天与昨天和明天相关。打乱的 K 折会把验证日放在训练日之间，模型实际上看到了每个测试日的“邻居”。TimeSeriesSplit 始终用过去训练、用其后的时间块验证。

# 7) Time series: shuffled K-fold vs TimeSeriesSplit vs the real future
from sklearn.model_selection import TimeSeriesSplit
from sklearn.linear_model import Ridge
from sklearn.ensemble import HistGradientBoostingRegressor
from sklearn.metrics import root_mean_squared_error
ts = pd.read_csv("daily_sales.csv", parse_dates=["date"])
f = pd.DataFrame({"t": np.arange(len(ts)), "weekend": (ts["date"].dt.dayofweek >= 5).astype(int),
                  "doy_sin": np.sin(2 * np.pi * ts["date"].dt.dayofyear / 365.25), "doy_cos": np.cos(2 * np.pi * ts["date"].dt.dayofyear / 365.25)})
past, future = ts["date"] < "2024-07-01", ts["date"] >= "2024-07-01"          # last 6 months = the real future
Xp, yp = f[past], ts.loc[past, "sales"]
gb = HistGradientBoostingRegressor(random_state=0)
for name, cv in [("KFold(shuffle=True)", KFold(5, shuffle=True, random_state=0)), ("TimeSeriesSplit(5, test_size=90)", TimeSeriesSplit(5, test_size=90))]:
    s = -cross_val_score(gb, Xp, yp, cv=cv, scoring="neg_root_mean_squared_error")
    print(f"{name:<33} estimated RMSE {s.mean():.1f}  (folds {np.round(s, 1)})")
gb.fit(Xp, yp)
print(f"real RMSE on Jul–Dec 2024                     {root_mean_squared_error(ts.loc[future, 'sales'], gb.predict(f[future])):.1f}")

Which Method Should You Use? 🧭कौन सा Method इस्तेमाल करें? 🧭该用哪种方法？🧭

Peeking: How Choosing Inflates Scores 👀Peeking: चुनने से Scores कैसे फूलते हैं 👀偷看：挑选如何抬高分数 👀

Every time you compare models on the same rows and keep the winner, you partly select for luck on those rows. We measured this on 30 simulated projects. Each compared 60 decision-tree settings using 200 validation rows and 200 test rows, and we checked the truth on 5,000 rows nobody used.

हर बार जब आप एक ही rows पर models की तुलना करके जीतने वाले को रखते हैं, तो आप कुछ हद तक उन rows पर क़िस्मत चुनते हैं। हमने इसे 30 simulated projects पर मापा। हर एक ने 200 validation rows और 200 test rows से 60 decision-tree settings की तुलना की, और हमने सच्चाई 5,000 ऐसी rows पर जाँची जिन्हें किसी ने इस्तेमाल नहीं किया।

每次在同一批行上比较模型并保留胜者，你都在一定程度上挑选了在这批行上的“运气”。我们在 30 个模拟项目上测量了这一点：每个项目用 200 行验证集和 200 行测试集比较 60 种决策树设置，再用 5,000 行谁也没用过的数据检验真实值。

# 8) Peeking: pick the best of many models on one held-out set, and that set's score becomes optimistic
from sklearn.datasets import make_classification
def one_project(seed, n_models=60):
    Xa, ya = make_classification(n_samples=6000, n_features=20, n_informative=5, flip_y=0.15, random_state=seed)
    Xtr, ytr, Xva, yva, Xte, yte = Xa[:600], ya[:600], Xa[600:800], ya[600:800], Xa[800:1000], ya[800:1000]
    Xnew, ynew = Xa[1000:], ya[1000:]                               # 5,000 rows the project never sees = the truth
    r = np.random.default_rng(seed); out = []
    for i in range(n_models):                                       # random decision-tree settings
        m = DecisionTreeClassifier(max_depth=int(r.integers(1, 15)), min_samples_leaf=int(r.integers(1, 40)),
                                   max_features=float(r.uniform(.2, 1)), random_state=i).fit(Xtr, ytr)
        out.append((accuracy_score(yva, m.predict(Xva)), accuracy_score(yte, m.predict(Xte)), accuracy_score(ynew, m.predict(Xnew))))
    return np.array(out)
gaps = {"best validation score": [], "test score of that model": [], "best TEST score (peeking)": []}
for seed in range(30):
    res = one_project(seed); b, b2 = res[:, 0].argmax(), res[:, 1].argmax()
    gaps["best validation score"].append(res[b, 0] - res[b, 2])
    gaps["test score of that model"].append(res[b, 1] - res[b, 2])
    gaps["best TEST score (peeking)"].append(res[b2, 1] - res[b2, 2])
print("average 'reported minus true accuracy' over 30 projects, 60 models each:")
for k, v in gaps.items():
    print(f"  {k:<28} {np.mean(v):+.3f}")

Nested Cross-Validation and Pipelines 🪆Nested Cross-Validation और Pipelines 🪆嵌套交叉验证与管道 🪆

GridSearchCV.best_score_ is the best of many scores on the same folds, so it carries the same selection bias. Nested CV puts the whole search inside each outer fold: the inner loop tunes, the outer fold — never seen by the search — scores. What it measures is the procedure "tune with this grid, then fit", not one fixed model.

GridSearchCV.best_score_ एक ही folds पर कई scores में सबसे अच्छा है, इसलिए उसमें वही selection bias है। Nested CV पूरी search को हर outer fold के अंदर रखता है: inner loop tune करता है, outer fold — जिसे search ने कभी नहीं देखा — score करता है। यह प्रक्रिया "इस grid से tune करो, फिर fit" को मापता है, किसी एक तय model को नहीं।

GridSearchCV.best_score_ 是同一批折上许多分数中的最大值，因此带有同样的选择偏差。嵌套交叉验证把整个搜索放进每一个外层折：内层负责调参，外层折 —— 搜索从未见过 —— 负责打分。它衡量的是“用这个网格调参再拟合”这一流程，而不是某一个固定模型。

# 9) Nested cross-validation: tune in an inner loop, measure in an outer loop
from sklearn.model_selection import GridSearchCV
from sklearn.svm import SVC
grid = {"svc__C": [0.1, 1, 10, 100], "svc__gamma": ["scale", 0.01, 0.001]}
inner, outer = StratifiedKFold(3, shuffle=True, random_state=42), StratifiedKFold(5, shuffle=True, random_state=42)
search = GridSearchCV(make_pipeline(StandardScaler(), SVC()), grid, cv=inner, scoring="accuracy")
gs = search.fit(X, y)
print(f"best_score_ of one search (tuned and scored on the same folds): {gs.best_score_:.4f}  best {gs.best_params_}")
nested = cross_val_score(search, X, y, cv=outer)                    # a fresh search inside every outer fold
print(f"nested CV (outer 5 folds never seen by the search)        : {nested.mean():.4f} ± {nested.std():.4f}")

Here the gap is small (0.979 vs 0.975): the grid has only 12 settings and the signal is strong. It grows with bigger grids, noisier data and fewer rows. Nested CV is nearly unbiased, not perfectly so, and the best parameters can differ between outer folds — which is itself useful information.

यहाँ अंतर छोटा है (0.979 बनाम 0.975): grid में सिर्फ़ 12 settings हैं और संकेत मज़बूत है। बड़े grids, शोर भरे data और कम rows के साथ यह बढ़ता है। Nested CV लगभग unbiased है, पूरी तरह नहीं, और सबसे अच्छे parameters outer folds के बीच अलग हो सकते हैं — जो ख़ुद काम की जानकारी है।

这里差距很小（0.979 vs 0.975）：网格只有 12 种设置，信号也很强。网格越大、数据越嘈杂、行数越少，差距越大。嵌套交叉验证近似无偏而非完全无偏，而且不同外层折选出的最佳参数可能不同 —— 这本身就是有用的信息。

Data Leakage — Measured 🚰Data Leakage — मापा गया 🚰数据泄漏 —— 实测 🚰

# 10) Three kinds of leakage, measured
from sklearn.feature_selection import SelectKBest, f_classif
from sklearn.pipeline import Pipeline
# a) target leakage: a column filled in AFTER the diagnosis
Xl = X.assign(referred_to_oncology=(y == 1) & (np.random.default_rng(0).random(len(y)) < 0.9))
print(f"a) with 'referred_to_oncology': CV accuracy {cross_val_score(model, Xl, y, cv=skf).mean():.3f} (without: {cross_val_score(model, X, y, cv=skf).mean():.3f})")
# b) preprocessing on all rows: pick 10 of 2,000 noise columns before CV vs inside CV
r = np.random.default_rng(1); Xn, yn = r.normal(size=(150, 2000)), r.integers(0, 2, 150)
keep = SelectKBest(f_classif, k=10).fit(Xn, yn).get_support()
wrong = cross_val_score(LogisticRegression(max_iter=1000), Xn[:, keep], yn, cv=skf).mean()
right = cross_val_score(Pipeline([("sel", SelectKBest(f_classif, k=10)), ("lr", LogisticRegression(max_iter=1000))]), Xn, yn, cv=skf).mean()
print(f"b) feature selection on pure noise: before the split {wrong:.3f} | inside a Pipeline {right:.3f} (truth 0.5)")
# c) the scaler on all rows (a small effect on this data, but the same habit)
sc_all = StandardScaler().fit(X)
leaky = cross_val_score(LogisticRegression(max_iter=5000), sc_all.transform(X), y, cv=skf).mean()
clean = cross_val_score(model, X, y, cv=skf).mean()
print(f"c) scaler fitted on all rows {leaky:.4f} | scaler inside the Pipeline {clean:.4f}")

Make the Practice Data 🐍Practice Data बनाएँ 🐍生成练习数据 🐍

Run this first. The breast-cancer data is real and ships with scikit-learn, so no download is needed. The daily sales series has a slow upward trend, a weekend bump, a yearly wave and day-to-day surprises that carry over to the next day.

इसे पहले चलाएँ। Breast-cancer data असली है और scikit-learn के साथ आता है, इसलिए download की ज़रूरत नहीं। Daily sales series में धीमा बढ़ता trend, weekend उछाल, सालाना लहर और रोज़ के आश्चर्य हैं जो अगले दिन तक चलते हैं।

请先运行它。乳腺癌数据是真实数据，随 scikit-learn 提供，无需下载。日销量序列包含缓慢上升的趋势、周末的增量、年度波动，以及会延续到第二天的日常波动。

# 0) Data: a real medical dataset (bundled with sklearn) and a seeded daily-sales series
import numpy as np, pandas as pd
from sklearn.datasets import load_breast_cancer
bc = load_breast_cancer(as_frame=True)
X, y = bc.data, 1 - bc.target                                   # 1 = malignant (the case we must catch)
print(f"breast cancer: {X.shape[0]} patients × {X.shape[1]} measurements, malignant {y.mean():.1%}")
rng = np.random.default_rng(5)
days = pd.date_range("2022-01-01", "2024-12-31", freq="D")
t = np.arange(len(days))
noise = np.zeros(len(t))
for i in range(1, len(t)):
    noise[i] = 0.8 * noise[i - 1] + rng.normal(0, 6)              # today's surprise carries into tomorrow
sales = 200 + 0.08 * t + 25 * (days.dayofweek >= 5) + 15 * np.sin(2 * np.pi * t / 365.25) + noise
ts = pd.DataFrame({"date": days, "sales": np.round(sales, 1)})
ts.to_csv("daily_sales.csv", index=False)
print(f"daily_sales.csv: {len(ts)} days, {days[0]:%d %b %Y} – {days[-1]:%d %b %Y}")

Golden Rules of Data Splitting 🏆Data Splitting के सुनहरे नियम 🏆数据划分黄金法则 🏆