Bias, Variance, Underfitting & Overfitting

The Story: Coach Arjun's Two Bad Archers

What Is Model Error? The Three-Part Decomposition

When a machine learning model makes predictions, its total expected error on unseen data can be mathematically decomposed into three independent sources. This is the Bias-Variance Decomposition.

The Four Scenarios — Archery Target Diagram

Bias — The Systematic Error

Variance — Sensitivity to the Training Set

The Bias-Variance Tradeoff — The Classic U-Curve

As model complexity increases, Bias² falls (the model becomes more expressive) but Variance rises (the model becomes more sensitive to training noise). Total error forms a U-shape. The minimum of that U is the sweet spot — the optimal model complexity.

Underfitting, Normal Fit & Overfitting — Polynomial Example

The clearest illustration uses polynomial regression on the same scatter data with three different model complexities — degree 1 (line), degree 3 (curve), and degree 10 (wiggly). The underlying true relationship is a gentle cubic.

Underfitting vs Normal Fit vs Overfitting — Side by Side

Learning Curves — The Diagnostic Tool

A learning curve plots training error and validation error against either training set size or number of training epochs. The shape of these two curves diagnoses whether you have a bias or variance problem.

Fixes for Underfitting (High Bias)

Fixes for Overfitting (High Variance)

Regularisation — The Most Powerful Overfitting Fix

Python — Diagnosing Bias & Variance with Learning Curves

import numpy as np
import matplotlib.pyplot as plt
from sklearn.pipeline       import Pipeline
from sklearn.preprocessing  import PolynomialFeatures, StandardScaler
from sklearn.linear_model   import Ridge
from sklearn.model_selection import learning_curve, validation_curve
from sklearn.datasets       import make_regression

# ── Generate data with a cubic relationship + noise ──────────
np.random.seed(42)
X_raw = np.random.uniform(-3, 3, (200, 1))
y     = 0.5*X_raw[:,0]**3 - X_raw[:,0]**2 + np.random.randn(200)*1.5

# ── Helper: plot learning curve ───────────────────────────────
def plot_learning_curve(model, X, y, title, ax, cv=5):
    sizes, train_s, val_s = learning_curve(
        model, X, y,
        train_sizes=np.linspace(0.1, 1.0, 10),
        cv=cv, scoring='neg_mean_squared_error'
    )
    train_err = -train_s.mean(axis=1)
    val_err   = -val_s.mean(axis=1)
    ax.plot(sizes, train_err, 'o-', label='Train MSE',      color='#f59e0b')
    ax.plot(sizes, val_err,   's--', label='Validation MSE', color='#6366f1')
    ax.set_title(title, fontsize=12)
    ax.set_xlabel('Training set size')
    ax.set_ylabel('MSE')
    ax.legend()
    ax.grid(True, alpha=0.3)

fig, axes = plt.subplots(1, 3, figsize=(15, 5))

# ── Three model complexities ──────────────────────────────────
models = [
    ('Underfit — Degree 1',
     Pipeline([('poly', PolynomialFeatures(1)),
                ('ridge', Ridge(alpha=1.0))])),

    ('Normal Fit — Degree 3',
     Pipeline([('poly', PolynomialFeatures(3)),
                ('ridge', Ridge(alpha=1.0))])),

    ('Overfit — Degree 15',
     Pipeline([('poly', PolynomialFeatures(15)),
                ('ridge', Ridge(alpha=0.0001))])),
]

for ax, (title, model) in zip(axes, models):
    plot_learning_curve(model, X_raw, y, title, ax)

plt.tight_layout()
plt.savefig('learning_curves.png', dpi=150)
plt.show()

Python — Validation Curve & Regularisation Tuning

import numpy as np
import matplotlib.pyplot as plt
from sklearn.datasets        import load_diabetes
from sklearn.linear_model    import Ridge, Lasso, RidgeCV
from sklearn.tree            import DecisionTreeRegressor
from sklearn.model_selection import (
    validation_curve, cross_val_score,
    train_test_split, KFold
)
from sklearn.preprocessing   import StandardScaler

X, y = load_diabetes(return_X_y=True)
scaler = StandardScaler()
X_sc   = scaler.fit_transform(X)

# ── 1. Validation curve: Decision Tree max_depth ─────────────
depths      = np.arange(1, 16)
train_s, val_s = validation_curve(
    DecisionTreeRegressor(), X_sc, y,
    param_name='max_depth', param_range=depths,
    scoring='neg_mean_squared_error', cv=5
)
plt.figure(figsize=(8, 4))
plt.plot(depths, -train_s.mean(1), 'o-', label='Train MSE',      color='#f59e0b')
plt.plot(depths, -val_s.mean(1),   's--', label='Validation MSE', color='#6366f1')
plt.axvline(x=depths[-val_s.mean(1).argmin()-1],
            color='#34d399', linestyle='--', label='Optimal depth')
plt.xlabel('max_depth')
plt.ylabel('MSE')
plt.title('Validation Curve — Bias-Variance vs Tree Depth')
plt.legend()
plt.show()

# ── 2. Ridge alpha tuning via RidgeCV ────────────────────────
alphas   = np.logspace(-4, 4, 50)
ridge_cv = RidgeCV(alphas=alphas, cv=5)
ridge_cv.fit(X_sc, y)
print(f"Best Ridge alpha (λ): {ridge_cv.alpha_:.4f}")

# ── 3. Cross-validation score comparison ─────────────────────
kf = KFold(n_splits=5, shuffle=True, random_state=42)
models_cv = {
    'Ridge (best λ)' : Ridge(alpha=ridge_cv.alpha_),
    'Decision Tree d=2 (underfit)': DecisionTreeRegressor(max_depth=2),
    'Decision Tree d=5 (good)'   : DecisionTreeRegressor(max_depth=5),
    'Decision Tree d=None (overfit)': DecisionTreeRegressor(),
}
print(f"\n{'Model':<35} {'CV MSE':>10}  {'Std':>8}")
print("-"*58)
for name, model in models_cv.items():
    scores = cross_val_score(
        model, X_sc, y,
        cv=kf, scoring='neg_mean_squared_error'
    )
    print(f"{name:<35} {-scores.mean():>10.1f}  {scores.std():>8.1f}")

Golden Rules