Loss Functions & Optimisation Objectives
Press Next → or use ← → arrow keys
The Archery Coach & the Scoreboard
It turns "how wrong was that prediction?" into a single number the optimiser can minimise. Lower loss means better predictions — so the whole of learning is just rolling that number downhill.
Loss vs Cost vs Objective
Three words that get used loosely but mean precise things. Getting them straight makes everything else click:
The loss measures fit; the penalty discourages needlessly complex models. Balancing the two is the heart of good training.
Mean Squared Error — The Regression Workhorse
Square each error, average them. Squaring kills the sign and punishes big misses far more than small ones — the loss curve is a smooth, convex bowl with a single global minimum.
Because errors are squared, a single wild miss can dominate the entire loss. Great when large errors truly matter — dangerous on heavy-tailed data.
MSE in Action: Predicting Rent
Five flats, predicted vs true monthly rent. Watch how one flat swamps the total:
| Flat | True (£) | Predicted (£) | Error | Squared Error |
|---|---|---|---|---|
| A | 1,200 | 1,150 | +50 | 2,500 |
| B | 850 | 900 | −50 | 2,500 |
| C | 2,000 | 1,700 | +300 | 90,000 |
| D | 950 | 940 | +10 | 100 |
| E | 1,500 | 1,520 | −20 | 400 |
Flat C's £300 miss contributes 90,000 of the 95,500 total. That's MSE's outlier vulnerability made concrete — the model will bend over backwards to fix C.
Binary Cross-Entropy — The Classification Standard
For yes/no problems, we don't measure distance — we measure surprise. Cross-entropy punishes confident wrong answers savagely: as a wrong prediction approaches certainty, the loss shoots toward infinity.
If y = 1, only −log(ŷ) counts; if y = 0, only −log(1−ŷ). The other half zeroes out.
log(0) = −∞. Always clip predictions to [ε, 1−ε] with ε = 1e-15 before computing cross-entropy.
Cross-Entropy in Action: Spam Filter
Four emails, each with the model's predicted probability of spam. The one confident mistake dominates:
| True | P(spam) | Active term | Loss | |
|---|---|---|---|---|
| E1 | SPAM (1) | 0.90 | −log(0.90) | 0.105 |
| E2 | HAM (0) | 0.05 | −log(0.95) | 0.051 |
| E3 | SPAM (1) | 0.10 | −log(0.10) | 2.303 |
| E4 | HAM (0) | 0.60 | −log(0.40) | 0.916 |
E3 was 90% sure a spam email was ham — and cross-entropy makes it pay dearly. That harsh signal is exactly what forces the model to fix its most dangerous mistakes first.
Categorical Cross-Entropy — Many Classes
When there are 3+ mutually exclusive classes, extend the idea. Labels are one-hot, predictions come from softmax, and only the true class's term survives.
Binary cross-entropy is categorical cross-entropy with exactly two classes. Same idea, one formula generalising the other.
The Unifying View: Maximum Likelihood
Why do these particular losses work? Because each one is what you get when you assume a noise distribution and maximise likelihood. Choosing a loss is choosing a model of the noise.
| Assumed noise | Negative log-likelihood | Loss you get |
|---|---|---|
| Gaussian N(ŷ, σ²) | Σ(yᵢ − ŷᵢ)² | MSE |
| Laplacian exp(−|y−ŷ|/b) | Σ|yᵢ − ŷᵢ| | MAE (L1 loss) |
| Bernoulli ŷʸ(1−ŷ)¹⁻ʸ | −Σ[y log ŷ + (1−y)log(1−ŷ)] | Binary Cross-Entropy |
| Categorical Πŷₖʸᵏ | −Σ yₖ log ŷₖ | Categorical Cross-Entropy |
Gaussian noise → MSE. Heavy-tailed Laplacian noise → MAE. Binary outcomes → cross-entropy. The loss isn't arbitrary — it encodes your assumption about how the data was generated.
Regularised Loss — Penalising Complexity
Add a penalty on the weights themselves and the optimiser is pushed toward simpler models that generalise better. Two classic penalties:
L2 is a bureaucrat taxing every weight's effort uniformly. L1 is a sculptor, carving unused weights away entirely until only the essential ones remain.
L1 vs L2 — and Tuning λ
| Property | L1 (Lasso) | L2 (Ridge) |
|---|---|---|
| Sparsity (zero weights) | Yes — exact zeros | No — only shrinks |
| Feature selection | Built-in | Not built-in |
| Correlated features | Picks one, drops rest | Spreads weight across all |
| Gradient at zero | Undefined (subgradient) | Smooth |
| Best when | Many irrelevant features | All features relevant |
There's a sweet spot between too much and too little regularisation. Find it with GridSearchCV or RidgeCV/LassoCV — never by intuition.
Gradient Descent — Rolling Downhill
The optimiser minimises the loss by repeatedly stepping in the direction that lowers it fastest — the negative gradient. Picture a ball rolling to the bottom of the loss valley:
For MSE, ∂L/∂ŷ = −2(y − ŷ). For cross-entropy with sigmoid/softmax, ∂L/∂ŷ ≈ ŷ − y — literally the prediction error. That elegance is why these losses train so smoothly.
Loss Functions From Scratch
Each loss is a few lines of NumPy — note the clip that keeps cross-entropy finite:
import numpy as np
def mse(y, p): return np.mean((y - p) ** 2)
def rmse(y, p): return np.sqrt(mse(y, p))
def binary_cross_entropy(y, p):
eps = 1e-15 # prevent log(0)
p = np.clip(p, eps, 1 - eps)
return -np.mean(y*np.log(p) + (1-y)*np.log(1-p))
def regularised_mse(y, p, w, lam=0.01, mode='l2'):
pen = lam*np.sum(w**2) if mode=='l2' else lam*np.sum(np.abs(w))
return mse(y, p) + pen
y = np.array([1200, 850, 2000, 950, 1500])
p = np.array([1150, 900, 1700, 940, 1520])
print(mse(y, p), rmse(y, p)) # 19100.0 138.2
yc = np.array([1, 0, 1, 0]); pc = np.array([.9, .05, .1, .6])
print(binary_cross_entropy(yc, pc)) # 0.843819,100 for the rent MSE, 0.8438 for the spam BCE — the code reproduces the worked examples exactly.
L1 vs L2 on Real Data
Two logistic-regression models on the Breast Cancer dataset — same data, different penalty. The trade-off is stark:
from sklearn.linear_model import LogisticRegression
lr_l2 = LogisticRegression(penalty='l2', C=1.0, max_iter=1000)
lr_l1 = LogisticRegression(penalty='l1', C=1.0, solver='liblinear')
# ...fit both on scaled features, then compare...| Model | Accuracy | BCE loss | Zero weights |
|---|---|---|---|
| L2 Ridge | 0.9737 | 0.0791 | 0 / 30 |
| L1 Lasso | 0.9649 | 0.1022 | 12 / 30 |
L2 keeps all 30 features and edges ahead on accuracy. L1 zeroes 12 of them — a small accuracy cost in exchange for a simpler, more interpretable model using only 18 features.
Choosing Your Loss Function
| Scenario | Loss function |
|---|---|
| Regression, Gaussian noise | MSE / RMSE |
| Regression, outlier-robust | MAE or Huber |
| Binary classification | Binary Cross-Entropy |
| Multi-class (mutually exclusive) | Categorical Cross-Entropy |
| Overfitting — many features | L1 penalty (Lasso) |
| Overfitting — correlated features | L2 penalty (Ridge) |
Continuous target with well-behaved errors → MSE. Probabilities → cross-entropy. Too many features → add an L1/L2 penalty. The task decides the loss.
6 Golden Rules
Pick the loss that matches your task and your noise, penalise complexity, and let gradient descent do the rest. Choose the objective well and the model learns the right thing.