Bias vs Variance Decoded

The Story: Vikram Needs to Predict Home Prices

The Data — Home Price vs Square Footage

Vikram plots all 21 homes. As square footage rises, price rises — but not perfectly. Real data always has noise. This is the dataset every model will be trained and tested on.

The Train / Test Split — The Foundation of All Evaluation

Before training any model, Vikram splits his 21 homes into two groups. Blue dots are the training set — the model learns from these. Orange dots are the test set — hidden during training, used only to measure real-world performance. This split is the key to revealing bias and variance.

The Complex Model — "The Memoriser" (Overfitting)

Vikram's first attempt is an extremely complex curve — high-degree polynomial that snakes through every single training point. It memorises the training data perfectly. Train error = 0.

Proving High Variance — The Same Model, a Different Training Set

Variance means: how much does the model's performance change when you train it on a different sample of the same data? Vikram shuffles his data and creates a second train/test split — different blue dots, different orange dots. He trains the same type of complex wiggly model again.

The Simple Model — "The Oversimplifier" (Underfitting)

Vikram's second attempt is the simplest possible model: a straight line. It cannot capture the slight upward curve in the data. Even on the training set, it makes consistent errors because a line simply cannot fit a curve.

Proving High Bias + Low Variance — The Linear Model on Two Splits

The Balanced Fit — "The Generalist" (The Sweet Spot)

Vikram's third model is a smooth, moderately curved function — complex enough to follow the general upward curve but not complex enough to memorise every noise fluctuation. This is the balanced fit.

The Complete Picture — All Three Models Side by Side

Why — The Mechanism Behind Each Behaviour

Python — Reproducing Vikram's Three Models

import numpy as np
from sklearn.pipeline        import Pipeline
from sklearn.preprocessing   import PolynomialFeatures
from sklearn.linear_model    import LinearRegression
from sklearn.model_selection import train_test_split, ShuffleSplit
from sklearn.metrics         import mean_squared_error

np.random.seed(42)

# ── Generate home price data (quadratic + noise) ─────────────
sqft  = np.random.uniform(600, 2500, 80)
price = 0.00004 * sqft ** 2 + 0.05 * sqft + np.random.randn(80) * 12
X     = sqft.reshape(-1, 1)
y     = price

# ── Three model pipelines ─────────────────────────────────────
models = {
    'Underfit (degree=1, linear)'    : Pipeline([
        ('poly',  PolynomialFeatures(1)),
        ('model', LinearRegression()),
    ]),
    'Balanced (degree=3)'            : Pipeline([
        ('poly',  PolynomialFeatures(3)),
        ('model', LinearRegression()),
    ]),
    'Overfit (degree=15, complex)'   : Pipeline([
        ('poly',  PolynomialFeatures(15)),
        ('model', LinearRegression()),
    ]),
}

# ── Run 2 different train/test splits (like the PDF slides) ──
splits = [42, 99]   # two different random seeds = two different training sets

print(f"{'Model':<32} {'Split':>6} {'Train MSE':>10} {'Test MSE':>10} {'Gap':>8}")
print("-" * 72)

for name, model in models.items():
    test_errors = []
    for i, seed in enumerate(splits, 1):
        X_tr, X_te, y_tr, y_te = train_test_split(
            X, y, test_size=0.25, random_state=seed
        )
        model.fit(X_tr, y_tr)
        tr_mse = mean_squared_error(y_tr, model.predict(X_tr))
        te_mse = mean_squared_error(y_te, model.predict(X_te))
        test_errors.append(te_mse)
        print(f"{name:<32} {'Split '+str(i):>6} {tr_mse:>10.1f} {te_mse:>10.1f} {te_mse-tr_mse:>8.1f}")

    variance_signal = abs(test_errors[1] - test_errors[0])
    print(f"{'':32} {'Variance (test diff):':>18} {variance_signal:>9.1f}")
    print()

Golden Rules