Backpropagation — Solved Numerical (2×2×2)
Press Next → or use ← → arrow keys
The Network
Two inputs feed two hidden neurons, which feed two outputs. Eight weights in all; sigmoid activation everywhere.
w₁–w₄ connect inputs to the hidden layer; w₅–w₈ connect hidden to outputs. With two outputs, the loss adds up the error from both Y₁ and Y₂.
The Givens
T₁ = 0.01, T₂ = 0.99
σ(z) = 1/(1 + e⁻ᶻ)
η = 0.5
w₅=0.40, w₆=0.45
w₇=0.50, w₈=0.55
Etotal = ½(T₁ − ŷ₁)² + ½(T₂ − ŷ₂)². Each output contributes its own squared error; backprop will send blame back from both.
Forward Pass — Hidden Layer
Weighted sum (net), then sigmoid. The net values below already fold in the hidden-layer bias:
outH1 = 0.5944 and outH2 = 0.5963 now feed both output neurons.
Forward Pass — Output Layer
Each output mixes both hidden activations through its own pair of weights, then squashes:
We predicted ŷ₁ = 0.7569 (target 0.01 — far too high) and ŷ₂ = 0.7685 (target 0.99 — a bit low). Both need correcting, in opposite directions.
Total Error
Score each output with ½(T − ŷ)², then add them up:
Y₁ is wildly off (0.76 vs 0.01), so it owns most of the error — and will drive the biggest weight changes in the backward pass.
Forward Pass — On the Network
Every value from the forward pass, in place:
The distance between each ŷ (green) and its target T (red) is what backprop now works to close, weight by weight.
Two Chain-Rule Patterns
Every weight's gradient is a product of local derivatives. Output weights need three factors; hidden weights need four.
For an output weight it's the hidden activation; for a first-layer weight it's the raw input x. That single fact anchors the last term of every chain.
Updating w₅ — An Output Weight
w₅ connects H₁ → Y₁. Its gradient is the 3-term chain: error signal × sigmoid slope × incoming activation.
ŷ₁ was far too high, so the gradient is positive and w₅ drops from 0.40 to 0.3592 — pulling Y₁'s output down toward its target of 0.01.
Updating w₁ — A Hidden Weight
w₁ connects x₁ → H₁, buried one layer deeper. Following the error along the Y₁ path gives a 4-term chain:
H₁ feeds both Y₁ and Y₂, so the full hidden gradient also adds the Y₂ path (through w₆). Here we follow the single Y₁ path the walkthrough demonstrates — the method is identical, just summed over outputs.
Every Weight Follows the Pattern
Two solved, six to go — but each is the very same recipe with different indices. Apply, then update all at once with η = 0.5:
| Weight | Type | Chain | Old → New |
|---|---|---|---|
| w₅ (H₁→Y₁) | Output | 3-term | 0.40 → 0.3592 |
| w₆ (H₁→Y₂) | Output | 3-term | same pattern |
| w₇, w₈ | Output | 3-term | same pattern |
| w₁ (x₁→H₁) | Hidden | 4-term | 0.15 → 0.1486 |
| w₂, w₃, w₄ | Hidden | 4-term | same pattern |
Never update a weight mid-calculation. Compute every gradient using the original weights, then apply all the updates together — that's one clean training step.
Python — Every Number Confirmed
import numpy as np
x1, x2 = 0.05, 0.10; T1, T2 = 0.01, 0.99; lr = 0.5
w5, w6, w7, w8 = 0.40, 0.45, 0.50, 0.55
sig = lambda z: 1/(1+np.exp(-z))
sigD = lambda z: sig(z)*(1-sig(z))
# forward (net values include the bias)
net_H1, net_H2 = 0.3825, 0.390
out_H1, out_H2 = sig(net_H1), sig(net_H2) # 0.5944, 0.5963
net_Y1 = w5*out_H1 + w7*out_H2
net_Y2 = w6*out_H1 + w8*out_H2
out_Y1, out_Y2 = sig(net_Y1), sig(net_Y2) # 0.7569, 0.7685
E_total = 0.5*(T1-out_Y1)**2 + 0.5*(T2-out_Y2)**2 # 0.3042
# backward — w5 (3-term chain)
grad_w5 = (out_Y1 - T1) * sigD(net_Y1) * out_H1 # 0.0817
w5_new = w5 - lr * grad_w5 # 0.3592
# backward — w1 (4-term chain, Y1 path)
grad_w1 = (out_Y1 - T1) * sigD(net_Y1) * w5 * x1 # 0.00275
w1_new = 0.15 - lr * grad_w1 # 0.1486Etotal = 0.3042, ∂E/∂w₅ = 0.0817 → 0.3592, and the w₁ chain all match the paper solution.
7 Golden Rules for Solving on Paper
Forward, error, 3-term output gradients, 4-term hidden gradients, simultaneous update. Scale the same five moves up and you have every deep network ever trained.