Machine Learning 📂 Supervised Learning · 6 of 19 203 min read

Bias vs Variance Decoded

Learn the Bias–Variance Tradeoff from scratch using a realistic Home Price vs Square Footage dataset. This tutorial visually and numerically explains why simple linear models underfit with high bias, why highly complex models overfit with high variance, and how balanced models achieve the best generalization. Includes train/test split intuition, multiple training split proofs, SVG diagrams, and complete Python code using Scikit-learn.

🌐 Language
Section 01

The Story: Vikram's Three Models and Coach Arjun's Two Archers

📖 Real-World Story
Three Models — Only One Gets Hired
Vikram is a data scientist at a property firm in Pune. His job is to predict a home's price from its size (square feet). He collects data on 21 homes. Bigger homes cost more, but the points do not sit on a perfect line — real data always has noise.

He tries three models: a straight line, a very wiggly curve and a smooth, gently bending curve. His manager asks one question: "Which model do I trust?"

Vikram's friend, Coach Arjun, trains archers. He says: "I have the same problem. Priya shoots tight groups — but always in the bottom-left corner. Ravi sprays arrows everywhere — sometimes near the centre, often not. I want an archer whose arrows are tight and centred."

Priya has bias. Ravi has variance. Vikram's straight line behaves like Priya, and his wiggly curve behaves like Ravi. This tutorial shows exactly why — with numbers you can measure yourself.
▶️
This Tutorial Has Six Simulators

Look for the interactive boxes: the Archery Lab (Section 04), Many Training Sets (05), The U-Curve, Computed (06), The Split Lottery (08), Learning Curves (09) and Taming a Wiggly Model (12). They train real models on random data in your browser, so every bias, variance and error you see is measured, not drawn by hand.

Section 02

Where Prediction Error Comes From

Every price in Vikram's data has two parts: the true pattern (what a home of that size is really worth on average) and noise (location, age, a pushy buyer — things the model cannot see).

The data
y = f(x) + ε
x = home size, y = observed price.
f(x) = the true pattern (unknown to us).
ε = noise with average 0 and variance σ².
The model
ŷ = f̂D(x)
We train a model on a training set D — a random sample of homes. The subscript D is important: a different sample gives a different model.

This is the key idea of the whole topic. Vikram's 21 homes are only one random sample. If he had collected 21 other homes, his model — and its predictions — would come out a bit different. Bias and variance describe how a model behaves across all the training sets it could have been given.

Part of the errorWhere it comes fromCan we reduce it?Archery picture
Bias²The model is too simple for the true pattern (wrong assumptions)Yes — more flexible modelPriya's group is centred in the wrong place
VarianceThe model changes a lot when the training sample changesYes — simpler model, penalty, more dataRavi's arrows land in a different place every time
Noise σ²Randomness in the data itselfNo — it is the floorA gust of wind no archer can control
Section 03

The Exact Definitions of Bias and Variance

Fix one home size, x₀ (say 2 300 sq ft). Now imagine training the same kind of model on many different training sets D₁, D₂, D₃, … Each model gives its own prediction for x₀. ED[ … ] means "the average over all those training sets".

Bias — how far the average prediction is from the truth
Bias(x₀) = ED[ f̂D(x₀) ] − f(x₀)
Average the predictions of all the models, then compare with the true pattern f(x₀) — not with one noisy price. If the models predict too low on average, the bias is negative; too high → positive. We usually look at Bias², so the sign does not matter.
Variance — how much the predictions spread out
Var(x₀) = ED[ ( f̂D(x₀) − ED[ f̂D(x₀) ] )² ]
For each model, take its prediction minus the average prediction, square it, and average. It measures how much the model changes when the training set changes. The true price does not appear in it at all.
The decomposition — why these two matter
E[ (y₀ − f̂D(x₀))² ] = Bias(x₀)² + Var(x₀) + σ²
The average squared error on a new home of size x₀ (averaged over training sets and over the noise in its price) splits exactly into three parts. Average over all home sizes and you get the model's expected test MSE.
🧮 Worked Example — Five Training Sets, One Home of 2 300 sq ft (true price f(x₀) = 90 lakh)
Predictions
Five models, each trained on a different sample of homes, predict: 78, 82, 80, 76, 84 lakh.
Average
E[ŷ] = (78 + 82 + 80 + 76 + 84) / 5 = 80
Bias
Bias = 80 − 90 = −10 (always too low) → Bias² = 100
Variance
Deviations from 80: −2, 2, 0, −4, 4. Squares: 4, 4, 0, 16, 16.
Variance = (4 + 4 + 0 + 16 + 16) / 5 = 8 (spread ≈ ±2.8)
Verdict
The models agree with each other (low variance) but are all wrong in the same direction (high bias). This is Priya — and Vikram's straight line.
⚠️
Four Common Mix-Ups

1. Variance is not the train–test gap. A big gap between training and test error is a symptom that often comes with high variance, but it is not the definition. Variance is the spread of predictions across training sets.
2. Variance is not the spread of the data. The scatter of prices around the true curve is the noise σ². A model can have tiny variance on very noisy data (a flat line barely moves).
3. Bias compares with f(x), not with y. The true pattern has no noise; one observed price does.
4. Bias and variance belong to a model type plus a data size (for example "a degree-10 polynomial trained on 20 homes"), not to one fitted model. One fitted model has an error; only many training sets have a bias and a variance.

Section 04

The Archery Picture — Four Targets

Think of each arrow as one model trained on one training set, and the bullseye as the true value f(x₀). Bias is the distance from the bullseye to the centre of the group. Variance is how widely the arrows spread around their own centre.

🎯 Bias vs Variance — Four Archery Targets
LOW VARIANCE HIGH VARIANCE LOW BIAS HIGH BIAS ✓ Low Bias + Low Variance IDEAL — The Sweet Spot Low Bias + High Variance Scattered around centre High Bias + Low Variance Consistent but off-target High Bias + High Variance Worst case — scattered off-target

Each dot is one model trained on a different sample of training data. The bullseye is the true target value. Green = ideal, Amber = overfitting, Blue = underfitting, Red = worst possible outcome.

Try It — Archery Lab

Set Priya's aim offset (bias) and Ravi's shaky hands (spread), then shoot. For any group of arrows, the average squared miss is exactly bias² + variance — the same split as above, just without noise.

🏹 Archery LabAverage squared miss = bias² + variance.
Archer:
aim offset (bias)
shaky hands (spread)
Section 05

Watch It Live — One Model Type, Many Training Sets

Now we do the definition for real. The dashed curve is the true pattern f(x) behind Vikram's homes. Each press draws a new random training set of homes (blue dots) and fits the chosen model (green). After many presses, the purple curve is the average model ED[f̂(x)].

The right panel zooms in on one home size (the purple dashed line — drag it on the chart). Every green dot is one model's prediction. The red arrow from the truth to the average is the bias. The shaded band is ± one standard deviation — its square is the variance.

🎲 Many Training SetsBias and variance, measured exactly as defined.
Model:
polynomial degree
homes per training set
noise
home size x₀
💡
What to Try

Add 50 training sets with the straight line: the curves hug each other, but their average never bends — high bias, low variance. Switch to degree 10 (the same training sets are refitted): the average curve now follows the truth, but single curves swing wildly — low bias, high variance. Then raise the number of homes to 100 and watch the degree-10 spread shrink.

Section 06

The Bias-Variance Tradeoff — The U-Curve, Measured

As the model becomes more flexible, bias² falls (it can follow the true curve) but variance rises (it also follows the noise of each training set). Their sum with the noise floor makes a U-shape. The bottom of the U is the sweet spot.

The chart below is not a sketch. For every degree it trains 300 models on 300 fresh training sets and measures bias², variance and the real error on new homes. Click a degree to see its numbers.

📈 The U-Curve, ComputedBias² + variance + noise = test error, for every complexity.
Homes per training set:
homes per training set
noise
— bias²   — variance   - - noise σ²   ━ sum ● measured test   - - train error
Few parameters (e.g. a line: 2)Right number (e.g. degree 2–3)Many parameters (e.g. degree 10: 11)
What it can drawOnly straight lines — cannot bendThe real curve, but not every bumpAlmost any shape — including the noise
Effect of a new training setLine barely movesCurve moves a littleCurve changes shape completely
ResultHigh bias · low varianceLow bias · low varianceLow bias · high variance
Section 07

Underfitting vs Good Fit vs Overfitting — Side by Side

The first rows below are measured with the simulator (20 homes per training set, noise σ = 10 lakh, so σ² = 100). Notice that training error keeps falling while test error does not.

PropertyUnderfittingGood fitOverfitting
Vikram's modelStraight line (degree 1)Gentle curve (degree 2)Wiggly curve (degree 10)
Bias² (measured)≈ 111 — high≈ 2≈ 0
Variance (measured)≈ 10≈ 15≈ 270 — high
Expected test error≈ 221≈ 118 — lowest≈ 370
Average training error≈ 200 (high)≈ 87≈ 45 (looks great!)
Train–test gapSmall — both badSmall — both goodLarge
More training data helps?No — bias staysA littleYes — variance falls
Decision tree examplemax_depth = 1 (stump)max_depth ≈ 3–6 (tune it)max_depth = None
Neural network exampleToo few neurons, heavy penaltyTuned size + early stoppingHuge network, tiny data, no penalty
✅
Why Training Error Is Below the Noise Floor

The noise floor is σ² = 100, yet the good model's training error is only ≈ 87 and the wiggly one's ≈ 45. A model always fits the homes it was trained on a little too well: with p parameters and n homes, the training error is roughly σ² × (1 − p/n) — for degree 10: 100 × (1 − 11/20) = 45. That is why training error alone can never tell you a model is good.

Section 08

Diagnosing From Train and Test Errors

In real life Vikram has only one dataset. He cannot draw 300 fresh training sets, so he cannot measure bias and variance directly. Instead he splits his homes into a training set (blue) and a test set (orange), and reads the errors as clues:

🔑 The Two Diagnostic Clues
Bias clue
Look at the training error (and compare it with the noise level you expect). High training error → the model cannot even fit what it was shown → likely high bias.
Variance clue
Look at the gap between test and training error, and at how much the test error jumps between different splits (cross-validation folds). Big gap or big jumps → likely high variance.
Goal
Low training error, a small gap, and a test error that stays steady across splits.
Careful
These are estimates from one small dataset. The test error of a single split — or the difference between two splits — is itself noisy. Use k-fold cross-validation and look at the average and the spread.

Try It — The Split Lottery

These are Vikram's 21 homes. Each press puts 14 random homes in training and 7 in testing. Press New split a few times for each model, then Run 200 splits to see how much the test error depends on luck.

🎫 The Split LotterySame 21 homes, different random splits.
Model:
polynomial degree
● train   ● test (dotted line = its error)   - - true pattern
Section 09

Learning Curves — What More Data Can and Cannot Fix

A learning curve plots training error and test (validation) error against the number of training examples. Its shape tells you which problem you have — and whether collecting more data is worth it.

📚 Learning CurvesMore data shrinks variance. It cannot shrink bias.
● train error   ● test error   ··· bias² + noise   - - noise floor
Shape of the curvesDiagnosisWill more data help?
Both curves meet quickly, far above the noise floorHigh bias (underfitting)No
Wide gap; test error falls slowly as data growsHigh variance (overfitting)Yes
Curves meet close to the noise floorGood fitOnly a little
Section 10

Fixes for Underfitting (High Bias)

💡
Diagnosis First

You have an underfitting problem if: training error is high, validation error is also high, and the gap between them is small. Collecting more data will not help. You need to give the model more capacity to learn.

📈
Increase Model Complexity
Use a higher-degree polynomial, a deeper decision tree, more neurons, or more layers. Switch from a linear model to a non-linear one (e.g. SVM with RBF kernel, Random Forest).
✓ Directly reduces bias
✗ Risk of introducing variance — monitor validation error
🛠️
Add More Features
Provide the model with richer information. Engineer interaction terms (x₁·x₂), polynomial features (x²), domain-specific ratios, or aggregations that capture the true underlying structure.
✓ Gives the model more signal to work with
✗ Irrelevant features add variance without reducing bias
🔧
Reduce Regularisation
If you applied L1/L2 regularisation, the penalty may be too strong, preventing the model from fitting even the training data. Lower λ (or raise C in sklearn) to give the model more freedom.
✓ Immediately loosens constraints on weights
✗ Too little regularisation → overfitting
⏱️
Train Longer
In neural networks and gradient boosting, early stopping or too few epochs/estimators can leave a model undertrained. Allow more iterations so the optimiser can reach a better minimum.
✓ Simple fix — no architecture change needed
✗ Watch for overfitting as epochs increase
🔄
Switch Algorithm Family
A fundamentally mismatched algorithm cannot be saved by tuning. If data has complex non-linear boundaries, move from Logistic Regression → Gradient Boosting or Neural Networks.
✓ Sometimes the only real fix
✗ New hyperparameters to tune from scratch
📉
Remove Excessive Feature Selection
If you removed too many features during preprocessing, you may have dropped important signal. Re-examine feature importances and restore features whose removal hurt training performance.
✓ Recovers lost information
✗ May re-introduce correlated / noisy features
Section 11

Fixes for Overfitting (High Variance)

⚠️
Diagnosis First

You have an overfitting problem if: training error is very low, validation error is significantly higher, and the gap is large. The model has memorised training data. Fixes focus on constraining the model or exposing it to more diverse data.

📦
Get More Training Data
The most reliable fix. More diverse examples force the model to learn generalised patterns rather than memorising specifics. Even synthetic data augmentation (flips, rotations, noise) can help in vision and NLP tasks.
Always try this first if possible
⚖️
Regularisation (L1 / L2)
Add a penalty to the loss function to keep weights small. L2 (Ridge): shrinks all weights — keeps all features with smaller influence. L1 (Lasso): forces some weights to exactly zero — automatic feature selection. λ controls the strength.
sklearn: C parameter in LogReg / alpha in Ridge, Lasso
✂️
Reduce Model Complexity
Lower polynomial degree, restrict tree depth (max_depth), reduce neurons / layers, or use fewer features. Directly reduces the model's capacity to memorise noise.
sklearn: max_depth, max_features, min_samples_leaf
🎲
Dropout (Neural Networks)
Randomly zero out a fraction of neurons during each training step. This forces the network to learn redundant representations and prevents co-adaptation of neurons. Typical rate: 0.2–0.5.
keras: Dropout(rate=0.3)
🛑
Early Stopping
Monitor validation error during training. Stop when validation error starts rising (even if training error continues to fall). Saves the model weights at the epoch of minimum validation loss.
sklearn: n_iter_no_change (GradientBoosting), early_stopping=True (MLP, HistGradientBoosting)
🌲
Ensemble Methods
Bagging (e.g. Random Forests) averages many high-variance models, dramatically reducing variance. Boosting builds shallow trees sequentially, keeping each one low-variance. Stacking combines diverse models to balance bias and variance simultaneously.
RandomForestClassifier, GradientBoostingClassifier
🧹
Feature Selection / PCA
Remove irrelevant or redundant features. High-dimensional data with many irrelevant columns gives the model too many ways to "explain" the training noise. SelectKBest, feature importances, or PCA all reduce dimensionality.
sklearn: SelectKBest, PCA, VarianceThreshold
🔁
Cross-Validation
Use k-fold cross-validation to get a reliable estimate of generalisation error during model selection and hyperparameter tuning. Never select hyperparameters based solely on a single train-test split — it can mislead.
sklearn: cross_val_score(model, X, y, cv=5)
Section 12

Regularisation, More Data and Bagging — Three Ways to Tame Variance

L2 Regularisation — Ridge
J(β) = Loss + λ · Σβⱼ²
Penalises large weights by adding their squared sum to the loss. All weights shrink toward zero but never reach it exactly. Good when all features are useful — keeps them all but constrains their influence.
L1 Regularisation — Lasso
J(β) = Loss + λ · Σ|βⱼ|
Penalises the absolute sum of weights. Creates sparsity — some weights are pushed to exactly zero, effectively removing features. Natural automatic feature selection.
🔍 How λ Controls the Bias-Variance Tradeoff
λ = 0
No penalty — ordinary least squares. Weights can grow as large as needed to fit training data. Risk of high variance on small datasets. Model memorises noise.
λ small
Mild constraint. Weights are slightly smaller. Good generalisation on large datasets. The sweet spot for most production models when tuned via cross-validation.
λ large
Strong constraint. Weights are forced near zero regardless of the data. Model predictions become near-constant — essentially the mean. High bias — underfitting territory.
Tuning
Always tune λ (alpha in Ridge / Lasso; C, which works like 1/λ, in LogisticRegression and SVM) via cross-validation. Try values on a logarithmic scale: [0.0001, 0.001, 0.01, 0.1, 1, 10, 100].
Use RidgeCV, LassoCV, or GridSearchCV to find the optimal value.

Try It — Taming a Wiggly Model

Start with a high-variance model trained on 20 homes: a degree-10 polynomial for the first two tabs, and a fully grown decision tree for bagging. Each tab tries one cure. The left chart shows 30 fitted models and their average; the right chart shows how bias² and variance move along the slider.

🛡️ Taming a Wiggly ModelThree ways to cut variance — and what each costs.
penalty λ (log scale)
Section 13

Python — Measuring Bias and Variance by Simulation

This code does exactly what the simulators do: it invents a true pattern, draws 2 000 training sets, trains the same model type on each, and applies the definitions. It also measures the real error on fresh homes, so you can check that bias² + variance + noise adds up. 

import numpy as np
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import FunctionTransformer, PolynomialFeatures
from sklearn.linear_model import LinearRegression

rng = np.random.default_rng(0)
NOISE = 10                                   # noise sd (₹ lakh) → σ² = 100

def true_price(sqft):                        # the real pattern f(x)
    u = (sqft - 1600) / 1000
    return 100 + 55*u + 35*u**2 + 10*u**3

def new_training_set(n=20):                  # one random sample of n homes
    sqft = 600 + 2000 * (np.arange(n) + rng.random(n)) / n   # spread over 600–2600 sq ft
    price = true_price(sqft) + rng.normal(0, NOISE, n)
    return sqft.reshape(-1, 1), price

def make_model(degree):                      # scale size to [-1, 1], then polynomial
    return make_pipeline(FunctionTransformer(lambda s: (s - 1600) / 1000),
                         PolynomialFeatures(degree), LinearRegression())

x0 = (600 + 2000 * (np.arange(60) + 0.5) / 60).reshape(-1, 1)   # homes to price
f0 = true_price(x0.ravel())
K = 2000                                     # number of training sets

print(f"{'Model':<14}{'Bias²':>8}{'Variance':>10}{'Noise':>7}{'Sum':>8}{'Measured':>10}{'Train':>8}")
print("-" * 65)
for degree in [1, 2, 10]:
    preds = np.empty((K, len(x0)))
    train = test = 0.0
    for k in range(K):
        X, y = new_training_set()
        model = make_model(degree).fit(X, y)
        preds[k] = model.predict(x0)
        train += np.mean((y - model.predict(X)) ** 2)
        y_new = f0 + rng.normal(0, NOISE, len(f0))          # fresh homes
        test += np.mean((y_new - preds[k]) ** 2)
    avg = preds.mean(axis=0)                               # E_D[ f̂(x) ]
    bias2 = np.mean((avg - f0) ** 2)                        # (E_D[f̂(x)] − f(x))²
    variance = np.mean(preds.var(axis=0))                   # E_D[(f̂(x) − E_D[f̂(x)])²]
    print(f"{'degree ' + str(degree):<14}{bias2:>8.1f}{variance:>10.1f}{NOISE**2:>7}"
          f"{bias2 + variance + NOISE**2:>8.1f}{test / K:>10.1f}{train / K:>8.1f}")
Output
Model Bias² Variance Noise Sum Measured Train ----------------------------------------------------------------- degree 1 111.0 10.3 100 221.3 222.7 200.4 degree 2 2.3 15.1 100 117.3 118.1 87.0 degree 10 0.0 261.6 100 361.6 361.6 45.4
🎯
Reading the Output

Degree 1: bias² ≈ 111 dominates → underfitting. Degree 10: variance ≈ 262 dominates → overfitting, even though its training error (45) is the best of the three. Degree 2: both small → the one to deploy. In every row, Sum ≈ Measured: the decomposition holds. You can only run this kind of check when you know the true pattern — with real data, use the clues from Sections 08 and 09.

Section 14

Python — Learning Curves, Validation Curves and Cross-Validation

With real data you cannot draw new training sets, so you use learning curves and cross-validation instead.

Learning curves for three model complexities

import numpy as np
import matplotlib.pyplot as plt
from sklearn.pipeline       import Pipeline
from sklearn.preprocessing  import PolynomialFeatures, StandardScaler
from sklearn.linear_model   import Ridge
from sklearn.model_selection import learning_curve, validation_curve
from sklearn.datasets       import make_regression

# ── Generate data with a cubic relationship + noise ──────────
np.random.seed(42)
X_raw = np.random.uniform(-3, 3, (200, 1))
y     = 0.5*X_raw[:,0]**3 - X_raw[:,0]**2 + np.random.randn(200)*1.5

# ── Helper: plot learning curve ───────────────────────────────
def plot_learning_curve(model, X, y, title, ax, cv=5):
    sizes, train_s, val_s = learning_curve(
        model, X, y,
        train_sizes=np.linspace(0.1, 1.0, 10),
        cv=cv, scoring='neg_mean_squared_error'
    )
    train_err = -train_s.mean(axis=1)
    val_err   = -val_s.mean(axis=1)
    ax.plot(sizes, train_err, 'o-', label='Train MSE',      color='#f59e0b')
    ax.plot(sizes, val_err,   's--', label='Validation MSE', color='#6366f1')
    ax.set_title(title, fontsize=12)
    ax.set_xlabel('Training set size')
    ax.set_ylabel('MSE')
    ax.legend()
    ax.grid(True, alpha=0.3)

fig, axes = plt.subplots(1, 3, figsize=(15, 5))

# ── Three model complexities ──────────────────────────────────
models = [
    ('Underfit — Degree 1',
     Pipeline([('poly', PolynomialFeatures(1)),
                ('ridge', Ridge(alpha=1.0))])),

    ('Normal Fit — Degree 3',
     Pipeline([('poly', PolynomialFeatures(3)),
                ('ridge', Ridge(alpha=1.0))])),

    ('Overfit — Degree 15',
     Pipeline([('poly', PolynomialFeatures(15)),
                ('ridge', Ridge(alpha=0.0001))])),
]

for ax, (title, model) in zip(axes, models):
    plot_learning_curve(model, X_raw, y, title, ax)

plt.tight_layout()
plt.savefig('learning_curves.png', dpi=150)
plt.show()

Validation curve, Ridge tuning and cross-validation

import numpy as np
import matplotlib.pyplot as plt
from sklearn.datasets        import load_diabetes
from sklearn.linear_model    import Ridge, Lasso, RidgeCV
from sklearn.tree            import DecisionTreeRegressor
from sklearn.model_selection import (
    validation_curve, cross_val_score,
    train_test_split, KFold
)
from sklearn.preprocessing   import StandardScaler

X, y = load_diabetes(return_X_y=True)
scaler = StandardScaler()
X_sc   = scaler.fit_transform(X)

# ── 1. Validation curve: Decision Tree max_depth ─────────────
depths      = np.arange(1, 16)
train_s, val_s = validation_curve(
    DecisionTreeRegressor(random_state=42), X_sc, y,
    param_name='max_depth', param_range=depths,
    scoring='neg_mean_squared_error', cv=5
)
plt.figure(figsize=(8, 4))
plt.plot(depths, -train_s.mean(1), 'o-', label='Train MSE',      color='#f59e0b')
plt.plot(depths, -val_s.mean(1),   's--', label='Validation MSE', color='#6366f1')
plt.axvline(x=depths[np.argmin(-val_s.mean(1))],   # depth with the lowest validation MSE
            color='#34d399', linestyle='--', label='Optimal depth')
plt.xlabel('max_depth')
plt.ylabel('MSE')
plt.title('Validation Curve — Bias-Variance vs Tree Depth')
plt.legend()
plt.show()

# ── 2. Ridge alpha tuning via RidgeCV ────────────────────────
alphas   = np.logspace(-4, 4, 50)
ridge_cv = RidgeCV(alphas=alphas, cv=5)
ridge_cv.fit(X_sc, y)
print(f"Best Ridge alpha (λ): {ridge_cv.alpha_:.4f}")

# ── 3. Cross-validation score comparison ─────────────────────
kf = KFold(n_splits=5, shuffle=True, random_state=42)
models_cv = {
    'Ridge (best λ)' : Ridge(alpha=ridge_cv.alpha_),
    'Decision Tree d=1 (underfit)': DecisionTreeRegressor(max_depth=1, random_state=42),
    'Decision Tree d=3 (good)'   : DecisionTreeRegressor(max_depth=3, random_state=42),
    'Decision Tree d=None (overfit)': DecisionTreeRegressor(random_state=42),
}
print(f"\n{'Model':<35} {'CV MSE':>10}  {'Std':>8}")
print("-"*58)
for name, model in models_cv.items():
    scores = cross_val_score(
        model, X_sc, y,
        cv=kf, scoring='neg_mean_squared_error'
    )
    print(f"{name:<35} {-scores.mean():>10.1f}  {scores.std():>8.1f}")
Output
Best Ridge alpha (λ): 0.1842 Model CV MSE Std ---------------------------------------------------------- Ridge (best λ) 3014.5 289.1 Decision Tree d=1 (underfit) 4714.0 120.2 Decision Tree d=3 (good) 3906.7 313.7 Decision Tree d=None (overfit) 6631.1 868.4
🎯
What the Output Tells Us

The depth-1 tree underfits: high CV MSE (4 714) but a small spread across folds (120) — consistently wrong, a bias clue. The unlimited tree overfits: the highest CV MSE (6 631) and by far the largest spread (868) — its score depends on which rows land in each fold, a variance clue. The depth-3 tree sits near the sweet spot (the validation curve bottoms out at depth 2–3), and the tuned Ridge model is best of all here. Note that the fold spread also includes how hard each fold happens to be, so compare spreads between models, not in isolation.

Section 15

Golden Rules

🎯 Bias, Variance & Model Fit — Key Rules
1
Bias and variance are defined over many training sets. Bias is how far the average model is from the true pattern. Variance is how much single models spread around that average. One fitted model has an error — it does not "have" a bias or a variance on its own.
2
Training error is a bias clue; the gap and the spread across folds are variance clues. High training error → suspect bias. Low training error with a much higher (or jumpy) test error → suspect variance. These are symptoms, measured on one dataset — not the definitions.
3
More data cures variance, not bias. A straight line trained on a million homes is still a straight line. A wiggly model trained on more homes calms down. Plot a learning curve before you spend money collecting data.
4
Near-zero training error is a warning sign. Real data has noise. A model that fits every training point has fitted the noise too. Expect a high, unstable test error.
5
The noise floor is real — do not chase it. No model can beat σ². If validation error has flattened close to the noise level, more complexity only adds variance.
6
Tune complexity with cross-validation, on a log scale. Search polynomial degree, tree depth or λ with GridSearchCV, RidgeCV or LassoCV. Report the average and the spread of the fold scores.
7
Averaging many models is the cheapest variance cure. Bagging and random forests average many high-variance models, so variance falls while bias barely moves. Boosting goes the other way: it builds many small, low-variance models and reduces bias step by step.