The Backpropagation Algorithm
Press Next → or use ← → arrow keys
The Story: Assigning the Blame
A network has thousands of weights. Which ones, and by how much, should change to reduce the loss? Backpropagation answers that for every weight at once — in a single backward sweep.
The Third Act of Training
Training is a loop of three moves. You've met the first two — backprop is what closes the circle and makes learning possible:
Backprop computes every gradient in a single backward sweep, no matter how deep the network. That efficiency — O(forward + backward), not O(weights) — is what made modern deep learning possible.
The Four Core Concepts
Backprop rests on four ideas. Nail these and the equations write themselves:
| Concept | Symbol | What it is |
|---|---|---|
| Computation graph | — | The network as a graph of operations — traversed forward to predict, backward to learn |
| Error signal (delta) | δ = ∂L/∂z | The "blame" at a neuron: how much the loss changes with its pre-activation z |
| Weight gradient | ∂L/∂W = δ · aᵀ | How to nudge each weight — the error signal times the incoming activation |
| Bias gradient | ∂L/∂b = δ | The error signal itself — the bias's input is always 1 |
Once you know the error signal δ at a neuron, both its weight and bias gradients follow immediately. The whole algorithm is really about computing δ at every neuron.
The Chain Rule — The Engine
Crossing a neuron's activation adds one more factor: ∂L/∂z = (∂L/∂a) × (∂a/∂z) = δ × σ′(z). That σ′(z) term is where saturation quietly kills gradients.
Two Passes Through One Graph
The same graph runs both ways: values flow forward (cyan) to make a prediction, then gradients flow backward (amber) to assign blame.
The backward pass reuses every activation a and pre-activation z from the forward pass. That's why frameworks cache them — you can't compute gradients without them.
The Four Key Equations
Backprop is really just these four formulas, applied layer by layer from the output backward:
Start with the output error (1). Push it back a layer with the transposed weights (2). At each layer, the two gradients (3) and (4) drop straight out of δ. Repeat to the input.
A Full Worked Example
A tiny network — 2 inputs → 1 hidden → 1 output, sigmoid activations, MSE loss — small enough to do entirely by hand.
y = 1.0
σ(z) = 1 / (1 + e⁻ᶻ)
L = ½(ŷ − y)²
b₁ = 0.1
W₂ = 0.9
b₂ = 0.1
First we run the input forward to get ŷ and the loss. Then we push the error backward to find every gradient — and finally update the weights.
Step 1 — The Forward Pass
A loss of 0.0550 says "clearly wrong, but not wildly." Now backprop finds exactly which knobs to turn — and how far — to shrink that number.
Step 2 — The Backward Pass
δ₂ = −0.0736 at the output becomes δ₁ = −0.0148 one layer back — multiplied down by W₂ and σ′(z₁). Stack many layers and this shrinking becomes the vanishing-gradient problem.
Step 3 — Update the Weights
Apply gradient descent with learning rate η = 0.5: θ ← θ − η · ∂L/∂θ. Every parameter takes one small step downhill:
| Parameter | Old value | Gradient | New value |
|---|---|---|---|
| W₂ | 0.9000 | −0.0488 | 0.9244 |
| b₂ | 0.1000 | −0.0736 | 0.1368 |
| W₁₁ | 0.4000 | −0.0074 | 0.4037 |
| W₁₂ | 0.6000 | −0.0118 | 0.6059 |
| b₁ | 0.1000 | −0.0148 | 0.1074 |
Because every gradient was negative, every weight nudged up — pushing the next prediction closer to 1.0. Repeat this loop thousands of times and the network learns.
Activations & Their Derivatives
Backprop needs the derivative of each activation — the σ′(z) factor in every δ. Here are the ones you'll use:
| Activation | Formula | Derivative | Issue |
|---|---|---|---|
| Sigmoid | 1/(1+e⁻ᶻ) | σ(z)(1−σ(z)) | Vanishing |
| Tanh | (eᶻ−e⁻ᶻ)/(eᶻ+e⁻ᶻ) | 1 − tanh²(z) | Less vanishing |
| ReLU | max(0, z) | 1 if z>0 else 0 | Dead neurons |
| Leaky ReLU | z if z>0 else 0.01z | 1 if z>0 else 0.01 | None |
Sigmoid's derivative peaks at just 0.25, so gradients shrink fast in deep nets. ReLU's is exactly 1 for positive inputs — which is precisely why it trains deep networks so much better.
Backprop From Scratch in NumPy
The forward and backward passes, side by side — the code mirrors the four equations exactly:
def sigmoid(z): return 1 / (1 + np.exp(-z))
def sigmoid_deriv(z): s = sigmoid(z); return s * (1 - s)
# Forward
z1 = W1 @ x + b1
a1 = sigmoid(z1)
z2 = W2 @ a1 + b2
y_hat = sigmoid(z2)
loss = 0.5 * (y_hat - y)**2
# Backward
delta2 = (y_hat - y) * sigmoid_deriv(z2) # δ₂ = -0.0736
dL_dW2 = delta2 @ a1.T # -0.0488
dL_db2 = delta2
dL_da1 = W2.T @ delta2
delta1 = dL_da1 * sigmoid_deriv(z1) # δ₁ = -0.0148
dL_dW1 = delta1 @ x.T # [-0.0074, -0.0118]
dL_db1 = delta1
# Update (lr = 0.5)
W2 -= 0.5 * dL_dW2; b2 -= 0.5 * dL_db2
W1 -= 0.5 * dL_dW1; b1 -= 0.5 * dL_db1δ₂ = −0.0736, δ₁ = −0.0148, and the weight gradients all match the worked example exactly.
Autograd — Backprop for Free
In practice you never hand-derive gradients. Frameworks record the forward operations and replay them backward automatically. One call does it all:
import torch
x = torch.tensor([[0.5], [0.8]])
y = torch.tensor([[1.0]])
W1 = torch.tensor([[0.4, 0.6]], requires_grad=True)
b1 = torch.tensor([[0.1]], requires_grad=True)
W2 = torch.tensor([[0.9]], requires_grad=True)
b2 = torch.tensor([[0.1]], requires_grad=True)
z1 = W1 @ x + b1
a1 = torch.sigmoid(z1)
z2 = W2 @ a1 + b2
y_hat = torch.sigmoid(z2)
loss = 0.5 * (y_hat - y)**2
loss.backward() # chain rule, applied automatically
print(W2.grad) # matches -0.0488 exactlyloss.backward() runs the exact four equations you just did by hand — and PyTorch's .grad values match the manual gradients to the last digit.
Common Pitfalls
Three things go wrong when gradients flow back through many layers. Each has a known fix:
Most training failures trace back to σ′(z) — either shrinking to nothing or zeroing out. Monitoring activation statistics catches these problems early.
7 Golden Rules
Forward to predict, loss to score, backward to learn. Backpropagation is the engine inside every deep network — from a 2-neuron toy to a trillion-parameter transformer.