Forward Propagation in Neural Networks
Press Next → or use ← → arrow keys
The Story: A Whisper Through Many Rooms
It's the process of passing an input through every layer — computing weighted sums and applying activations — to produce a final prediction. One direction, input to output.
Pure, One-Directional Arithmetic
During the forward pass, nothing learns. The weights are fixed; the network simply computes. Every prediction is a deterministic chain of multiply-add-activate steps.
Forward propagation produces the prediction. Only afterward does backpropagation measure the error and adjust the weights. This deck is entirely about that first, forward half.
The Four Core Operations
Every forward pass — from a single neuron to GPT — is built from just these four moves:
Affine then activation, over and over, then a softmax at the end. Master these four and you understand the forward pass of any feedforward network.
Operation 1: The Affine Transform
The workhorse step. Multiply the inputs by a weight matrix, add a bias vector — a linear reshaping of the data.
Stacking affine transforms only ever produces another affine transform. That's why every hidden layer must follow it with a non-linear activation.
Operation 2: The Activation
After the affine step, each value passes through a non-linear function a = f(z). The choice of f shapes what the network can learn:
| Function | Formula | Range | Note |
|---|---|---|---|
| ReLU | max(0, z) | [0, ∞) | Fast; risk of dead neurons |
| Sigmoid | 1/(1 + e⁻ᶻ) | (0, 1) | Vanishing gradient |
| Tanh | (eᶻ−e⁻ᶻ)/(eᶻ+e⁻ᶻ) | (−1, 1) | Zero-centred |
| Softmax | eᶻⁱ / Σeᶻʲ | probabilities | Output layer only |
Non-linearity is the reason depth matters. Remove f and the whole network collapses into a single linear layer, no matter how many you stack.
The Computation Graph
Zoom out and the forward pass is a graph: inputs on the left, each layer applying affine + activation, flowing rightward to the output. Here's a 2 → 2 → 2 network:
Each connection carries a weight; each node applies its activation. Follow the arrows left to right and you've computed the whole network's prediction.
Numerical 1: A Single Neuron
The smallest possible forward pass. Inputs x = [2, 3], weights W = [0.5, −0.4], bias b = 1, ReLU activation.
Multiply, add the bias, squash. Because 0.8 is positive, ReLU passes it straight through. A network is just thousands of these running in parallel and in sequence.
Numerical 2: A Full 2-Layer Network
Now a complete network: Input (2) → Hidden (2, ReLU) → Output (2, Softmax). Here's everything we're given:
W¹ = [[0.1, 0.2], [0.3, 0.4]]
b¹ = [0, 0]
b² = [0, 0]
output activation = Softmax
We'll push x through Layer 1 (affine → ReLU), then Layer 2 (affine → softmax), and read off a class probability. Next slide runs the numbers.
Running the Numbers
The network is more confident in class 1 (66.6%) than class 0 (33.4%). That single number is the end product of the entire forward pass — pure arithmetic, start to finish.
The Matrix View — One Layer at Once
We never loop over neurons one at a time. A single matrix multiply computes an entire layer, and a batch of inputs flows through together:
| Quantity | Shape | Meaning |
|---|---|---|
| Wˡ | (nₗ × nₗ₋₁) | weights: this layer's neurons × previous layer's |
| Aˡ⁻¹ | (nₗ₋₁ × m) | inputs for a batch of m examples |
| bˡ | (nₗ × 1) | bias, broadcast across the batch |
| Aˡ | (nₗ × m) | this layer's activations for all m examples |
Expressing a layer as one matrix multiply lets hardware process thousands of neurons and hundreds of examples in parallel — the reason modern networks train at all.
The Forward Pass in NumPy
The whole 2-layer example is a handful of lines — two matrix multiplies and two activations:
import numpy as np
x = np.array([1, 2], dtype=float)
W1 = np.array([[0.1, 0.2], [0.3, 0.4]])
b1 = np.zeros(2)
W2 = np.array([[0.5, -0.3], [-0.1, 0.6]])
b2 = np.zeros(2)
def relu(z): return np.maximum(0, z)
def softmax(z):
e = np.exp(z - np.max(z)) # stability trick
return e / e.sum()
z1 = W1 @ x + b1 # affine, layer 1
a1 = relu(z1) # activation
z2 = W2 @ a1 + b2 # affine, layer 2
y_hat = softmax(z2) # output probabilities
print(y_hat) # [0.334 0.666]
print("Class", np.argmax(y_hat)) # Class 1Each line is one operation from the walkthrough. @ is matrix multiply — the vectorized affine step in action.
Forward vs Backward — Two Halves
Forward propagation is only half the training loop. It's worth seeing exactly where it ends and learning begins:
| Forward propagation | Backpropagation | |
|---|---|---|
| Direction | Input → output | Output → input |
| Purpose | Produce a prediction | Compute gradients of the error |
| Weights | Unchanged | Updated |
| Maths | Matrix multiply + activation | Chain rule of derivatives |
| At test time | Runs alone | Not used |
A deployed model only runs forward propagation — identical to training, minus Dropout and with BatchNorm frozen. Backprop exists purely to teach the weights; once learned, the forward pass does the work.
6 Golden Rules
Affine, activate, repeat, softmax. From one neuron to a transformer, forward propagation is the same clean flow — and it's exactly what runs every time a model makes a prediction.