✍️

Backpropagation Solved Step by Step

One tiny network, one training step, every single number worked out by hand. Follow the error from the output all the way back to the first weight — the exact calculation a neural network runs billions of times.

2×2×1 Network Forward Pass Backward Pass Verified in Python

Press Next → or use ← → arrow keys

Section 01

The Network — Read the Diagram

Two inputs, two hidden neurons, one output. Every arrow is a weight; both layers use the sigmoid activation.

🗺️

Six weights to learn

Four weights feed the hidden layer, two feed the output. Our whole job is to nudge all six so the prediction moves closer to the target — and backprop tells us exactly how.

Section 01

The Givens

Everything we start with — inputs, target, weights, and the two functions:

📥 INPUTS & TARGET

          x₁ = 0.35,   x₂ = 0.70

          y = 1.0  (target)

          σ(z) = 1 / (1 + e⁻ᶻ)

          L = ½(ŷ − y)²

⚖️ INITIAL WEIGHTS

          w₁₁ = 0.2,   w₂₁ = 0.2

          w₁₂ = 0.3,   w₂₂ = 0.3

          w₁₃ = 0.3  (h₁→o₃)

          w₂₃ = 0.9  (h₂→o₃)

🧮

The plan

Run forward to get the prediction and loss, push the error backward for all six gradients, then update every weight with learning rate η = 0.5.

Section 02 · Forward

Forward Pass — Hidden Layer

Each hidden neuron takes its weighted sum, then squashes it with sigmoid:

1

h₁ sum: z_h1 = (0.2)(0.35) + (0.2)(0.70) = 0.070 + 0.140 = 0.2100

2

h₁ activation: a_h1 = σ(0.2100) = 0.5523

3

h₂ sum: z_h2 = (0.3)(0.35) + (0.3)(0.70) = 0.105 + 0.210 = 0.3150

4

h₂ activation: a_h2 = σ(0.3150) = 0.5781

📤

Two hidden activations, ready to combine

a_h1 = 0.5523 and a_h2 = 0.5781 now become the inputs to the output neuron.

Section 02 · Forward

Forward Pass — Output & Loss

The output neuron combines the two hidden activations, squashes once more, and we score the result:

1

o₃ sum: z_o3 = (0.3)(0.5523) + (0.9)(0.5781) = 0.1657 + 0.5203 = 0.6860

2

Prediction: ŷ = σ(0.6860) = 0.6651

3

Loss: L = ½(0.6651 − 1.0)² = ½(−0.3349)² = 0.0561

🎯

Predicted 0.6651, wanted 1.0

The network undershot. The loss of 0.0561 is the gap we now carry backward to fix — every weight will learn its share of the blame.

Section 03 · Backward

Backward — The Output Error δ

Backprop starts at the output. The error signal δ combines how the loss changes with the prediction and how the sigmoid bends there:

1

Loss derivative: ∂L/∂ŷ = ŷ − y = 0.6651 − 1.0 = −0.3349

2

Sigmoid slope: σ′(z_o3) = a_o3(1 − a_o3) = 0.6651 × 0.3349 = 0.2228

3

Output delta: δ_o3 = (−0.3349) × 0.2228 = −0.074617

🔑

δ is the reusable "blame"

Every gradient in the output layer, and every δ further back, is built from this one number. Compute δ once, reuse it everywhere.

Section 03 · Backward

Backward — Output Weight Gradients

Each output-weight gradient is simply δ_o3 times the activation that fed into it:

1

w₁₃ (from h₁): ∂L/∂w₁₃ = δ_o3 × a_h1 = −0.074617 × 0.5523 = −0.041212

2

w₂₃ (from h₂): ∂L/∂w₂₃ = δ_o3 × a_h2 = −0.074617 × 0.5781 = −0.043140

💡

Gradient = local error × incoming signal

A weight's gradient is always the error signal at its destination multiplied by the activation at its source. That single rule powers the entire backward pass.

Section 03 · Backward

Backward — Propagate to the Hidden Layer

Send δ_o3 back through the output weights, then multiply by each hidden neuron's sigmoid slope:

1

Into h₁: ∂L/∂a_h1 = δ_o3 × w₁₃ = −0.074617 × 0.3 = −0.022385; σ′(z_h1) = 0.2473 → δ_h1 = −0.005536

2

Into h₂: ∂L/∂a_h2 = δ_o3 × w₂₃ = −0.074617 × 0.9 = −0.067155; σ′(z_h2) = 0.2439 → δ_h2 = −0.016380

🔎

h₂ gets more blame — it had the bigger weight

Because w₂₃ = 0.9 versus w₁₃ = 0.3, more of the output error flows into h₂ (δ_h2 is three times δ_h1). Blame is shared in proportion to influence.

Section 03 · Backward

Backward — Input Weight Gradients

The last four gradients follow the same rule: each hidden δ times the input feeding that weight.

Weight	δ × input	Gradient
w₁₁ (x₁→h₁)	δ_h1 × 0.35	−0.001938
w₂₁ (x₂→h₁)	δ_h1 × 0.70	−0.003875
w₁₂ (x₁→h₂)	δ_h2 × 0.35	−0.005733
w₂₂ (x₂→h₂)	δ_h2 × 0.70	−0.011466

✅

All six gradients are now in hand

Two at the output, four at the input — every weight in the network has been assigned its exact share of the error. Time to update.

Section 03 · Backward

The Whole Picture at a Glance

Forward values in green, the error signals δ that flowed backward in amber:

🌊

Error shrinks as it travels back

δ starts at −0.0746 at the output and dwindles to −0.0055 and −0.0164 at the hidden layer. Multiply many such shrinking steps and you get the vanishing-gradient problem.

Section 04 · Update

Weight Update — Before & After

Apply w_new = w_old − η · gradient with η = 0.5. Every weight ticks upward:

Weight	Old	Gradient	New	Δ
w₁₁	0.2000	−0.001938	0.2010	+0.0010
w₂₁	0.2000	−0.003875	0.2019	+0.0019
w₁₂	0.3000	−0.005733	0.3029	+0.0029
w₂₂	0.3000	−0.011466	0.3057	+0.0057
w₁₃	0.3000	−0.041212	0.3206	+0.0206
w₂₃	0.9000	−0.043140	0.9216	+0.0216

🔁

That's one complete training step

The output weights (w₁₃, w₂₃) move most — they're closest to the error. Repeat this whole loop thousands of times and the loss steadily falls.

Section 04 · Update

Why Every Weight Increased

🧭 GRADIENT DESCENT DID ITS JOB

The prediction (0.6651) fell below the target (1.0), so the network needs a higher output. Every gradient came out negative — and subtracting a negative adds. So all six weights grew, pushing the next prediction upward toward 1.0. The maths and the intuition agree perfectly.

0.6651Prediction this step

1.0Target — we need to climb

↑ ×6All six weights increased

🎯

Sanity check that always works

If the prediction is too low, weights should rise; too high, they should fall. When the signs line up like this, your backprop is almost certainly correct.

Section 05

Python — Every Number Confirmed

The same calculation in a dozen lines. Run it and the outputs match our hand-work to four decimals:

import numpy as np
x1, x2 = 0.35, 0.70; y = 1.0; lr = 0.5
w11, w21, w12, w22 = 0.2, 0.2, 0.3, 0.3
w13, w23 = 0.3, 0.9
sig  = lambda z: 1/(1+np.exp(-z))
sigD = lambda z: sig(z)*(1-sig(z))

# forward
z_h1 = w11*x1 + w21*x2; a_h1 = sig(z_h1)   # 0.2100 → 0.5523
z_h2 = w12*x1 + w22*x2; a_h2 = sig(z_h2)   # 0.3150 → 0.5781
z_o3 = w13*a_h1 + w23*a_h2; a_o3 = sig(z_o3) # 0.6860 → 0.6651

# backward
d_o3 = (a_o3 - y) * sigD(z_o3)              # -0.074617
d_h1 = (d_o3 * w13) * sigD(z_h1)            # -0.005536
d_h2 = (d_o3 * w23) * sigD(z_h2)            # -0.016380

# update every weight: w -= lr * (delta * input)
w13 -= lr * d_o3 * a_h1                     # 0.3 → 0.3206
w23 -= lr * d_o3 * a_h2                     # 0.9 → 0.9216

✅

Hand-maths and code agree

δ_o3 = −0.074617, the hidden deltas, and every updated weight land exactly on the numbers we derived. That's your gradient check.

Section 06

The 8-Step Recipe

Paper-exam cheat-sheet — "ZASA-ΔWWU"

1Z — compute z (weighted sum) for every hidden neuron.

2A — apply the activation to get a.

3S·A — repeat the sum-and-activate for every layer up to the output.

4Compute the loss from the prediction and target.

5Δ — start backprop: find the output error signal δ.

6W — compute the output-layer weight gradients (δ × activation).

7W — propagate δ backward and get the hidden-layer gradients.

8U — update all weights simultaneously with w − η·gradient.

🚀

You can now solve any backprop by hand

Every deep network — from this 2×2×1 toy to a giant transformer — runs exactly these eight steps. The only difference is scale.