Loss Functions & Optimisation Objectives in Machine Learning

The Story That Explains Loss Functions

A loss function (also called a cost function or objective function) is a mathematical formula that takes the model's prediction ŷ and the true label y, and returns a single non-negative number representing how wrong the prediction is. The model's entire goal during training is to minimise this number.

Mean Squared Error — The Regression Workhorse

The Formula

🔨 Numerical Example 1 — MSE Step by Step

A model predicts apartment rents (£/month) for five flats. Let's compute the MSE manually.

When to Use MSE

Binary Cross-Entropy — The Classification Standard

The Formula

🔨 Numerical Example 2 — Cross-Entropy Step by Step

A spam classifier assigns probabilities to 4 emails. We compute the binary cross-entropy for each and then the average cost.

Maximum Likelihood Estimation — The Unifying View

You may wonder why MSE and cross-entropy look so different, yet both "work." The answer is that they are both instances of the same principle: Maximum Likelihood Estimation.

Regularised Loss — L1 and L2 Penalties

A model that perfectly minimises training loss often overfits — it memorises the training data including noise, and fails on new examples. Regularisation adds a penalty term to the objective that discourages the model from growing too complex.

Gradient Descent — How the Optimiser Actually Learns

Knowing what to minimise is only half the story. The optimiser is the algorithm that actually adjusts the model's weights to reduce the loss. The most fundamental optimiser is gradient descent.

Python Implementation — Full Working Code

Below is a complete example computing MSE, binary cross-entropy, and regularised loss from scratch in NumPy, then training a logistic regression model with both L1 and L2 regularisation using scikit-learn.

Part A — Loss Functions from Scratch

import numpy as np

# ── Mean Squared Error ──────────────────────────────────
def mse(y_true, y_pred):
    return np.mean((y_true - y_pred) ** 2)

# ── Root Mean Squared Error ─────────────────────────────
def rmse(y_true, y_pred):
    return np.sqrt(mse(y_true, y_pred))

# ── Binary Cross-Entropy ────────────────────────────────
def binary_cross_entropy(y_true, y_pred):
    eps = 1e-15  # prevent log(0)
    y_pred = np.clip(y_pred, eps, 1 - eps)
    return -np.mean(
        y_true * np.log(y_pred) + (1 - y_true) * np.log(1 - y_pred)
    )

# ── Regularised MSE ─────────────────────────────────────
def regularised_mse(y_true, y_pred, weights, lam=0.01, mode='l2'):
    loss = mse(y_true, y_pred)
    if mode == 'l2':
        penalty = lam * np.sum(weights ** 2)
    elif mode == 'l1':
        penalty = lam * np.sum(np.abs(weights))
    else:
        penalty = 0
    return loss + penalty

# ── Demo ─────────────────────────────────────────────────
y_true_reg  = np.array([1200, 850, 2000, 950, 1500])
y_pred_reg  = np.array([1150, 900, 1700, 940, 1520])
weights     = np.array([0.5, -0.3, 1.2, 0.8])

print(f"MSE:         {mse(y_true_reg, y_pred_reg):,.1f}")
print(f"RMSE:        £{rmse(y_true_reg, y_pred_reg):.1f}")

y_true_cls = np.array([1, 0, 1, 0])
y_pred_cls = np.array([0.90, 0.05, 0.10, 0.60])
print(f"BCE:         {binary_cross_entropy(y_true_cls, y_pred_cls):.4f}")
print(f"Reg MSE L2:  {regularised_mse(y_true_reg, y_pred_reg, weights, 0.01, 'l2'):,.2f}")
print(f"Reg MSE L1:  {regularised_mse(y_true_reg, y_pred_reg, weights, 0.01, 'l1'):,.2f}")

Part B — Scikit-learn: L1 vs L2 on Breast Cancer Dataset

from sklearn.datasets import load_breast_cancer
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
from sklearn.metrics import log_loss, accuracy_score
import numpy as np

# Load data
data = load_breast_cancer()
X, y = data.data, data.target

X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42
)
scaler = StandardScaler()
X_train = scaler.fit_transform(X_train)
X_test  = scaler.transform(X_test)

# L2 Regularised Logistic Regression (default)
lr_l2 = LogisticRegression(penalty='l2', C=1.0, max_iter=1000)
lr_l2.fit(X_train, y_train)

# L1 Regularised Logistic Regression
lr_l1 = LogisticRegression(penalty='l1', C=1.0, solver='liblinear')
lr_l1.fit(X_train, y_train)

for name, model in [('L2 Ridge', lr_l2), ('L1 Lasso', lr_l1)]:
    probs = model.predict_proba(X_test)
    acc   = accuracy_score(y_test, model.predict(X_test))
    bce   = log_loss(y_test, probs)
    zeros = np.sum(model.coef_ == 0)
    print(f"{name} | Accuracy: {acc:.4f} | BCE Loss: {bce:.4f} | Zero weights: {zeros}/30")

Quick Reference — Choosing Your Loss Function

Golden Rules