Recurrent Neural Networks
Press Next → or use ← → arrow keys
The World Is Full of Sequences
An RNN loops its own output back in as input, maintaining a hidden state that acts as memory. Each new step updates that memory with the latest input — so context accumulates across the sequence.
Why a Standard Network Can't Do This
A feedforward network treats every input independently and expects a fixed size. Sequences break all three assumptions:
One cell, reused at every time step, with a hidden state passed along — handles any length, respects order, and shares one set of weights across the whole sequence.
The Recurrent Loop
At its heart, an RNN is a single cell with a loop: its hidden state feeds back into itself at the next step. That loop is the memory.
The hidden state hₜ produced at one step is fed back in at the next — so the cell always sees both the new input and a summary of everything before it.
Unrolled Through Time
Unfold that loop and the RNN becomes a chain — one copy of the same cell per time step, passing the hidden state down the line:
It's not three different cells — it's one cell reused three times. Sharing the weights across time is what lets an RNN handle any length and generalise a pattern no matter where it appears.
The Mathematics
Two equations run at every time step. The first updates the memory; the second reads out a prediction:
| Term | Shape | Role |
|---|---|---|
| Wₓ | hidden × input | How the new input enters the memory |
| Wₕ | hidden × hidden | How the old memory carries forward |
| tanh | → (−1, 1) | Squashes the state, keeping it bounded |
The magic is the sum Wₓxₜ + Wₕhₜ₋₁: the present and the past are blended into one new state, then squashed by tanh.
The Four Sequence Patterns
By choosing where inputs and outputs attach, one RNN handles four very different jobs:
| Pattern | Shape | Example task |
|---|---|---|
| One-to-one | 1 in → 1 out | Plain image classification |
| One-to-many | 1 in → sequence out | Image captioning, music generation |
| Many-to-one | sequence in → 1 out | Sentiment analysis, time-series forecast |
| Many-to-many | sequence in → sequence out | Machine translation, speech recognition |
Nothing changes inside the RNN — only which time steps you feed inputs to and read outputs from. That flexibility is why RNNs fit so many domains.
The Vanishing Gradient Problem
With a factor of 0.9 per step, 0.9¹⁰⁰ ≈ 0.000027 — effectively zero. Above 1 the opposite happens: gradients explode to NaN. Both wreck vanilla-RNN training on long sequences.
Backpropagation Through Time
RNNs train with BPTT — unroll the network across time, then backprop through the whole chain:
Because W is reused at every step, its gradient is the sum of contributions from all steps — which is exactly why long chains cause vanishing or exploding gradients.
LSTM — Long Short-Term Memory
The cell state flows forward with only pointwise multiplication by the forget gate — no squashing. That near-uninterrupted path is how LSTMs largely solve the vanishing gradient.
GRU — Gated Recurrent Unit
The GRU is a streamlined LSTM: two gates instead of three, no separate cell state, roughly 25% fewer parameters — and usually just as good.
Fewer gates means fewer parameters and quicker training. For most tasks a GRU matches an LSTM — which is why it's a great default.
Vanilla RNN vs LSTM vs GRU
| Property | Vanilla RNN | LSTM | GRU |
|---|---|---|---|
| Memory | Hidden state only | Cell + hidden state | Gated hidden state |
| Gates | 0 | 3 | 2 |
| Vanishing gradient | Severe | Largely solved | Largely solved |
| Parameters | Fewest | Most | ~25% less than LSTM |
| Training speed | Fastest | Slowest | Fast |
| Long sequences | Fails | Excellent | Very good |
Never reach for the vanilla RNN beyond ~20 steps. Start with a GRU; move to an LSTM if you need its extra capacity on very long or complex sequences.
Code — Vanilla RNN & PyTorch LSTM
The vanilla RNN cell is four lines of NumPy; a real LSTM classifier is a short PyTorch module:
# Vanilla RNN cell (NumPy) — one time step
h[t] = np.tanh(Wx @ x[t] + Wh @ h[t-1] + bh) # hidden state
y[t] = Wy @ h[t] + by # logits
p[t] = np.exp(y[t]) / np.sum(np.exp(y[t])) # softmax
# LSTM sentiment classifier (PyTorch)
class SentimentLSTM(nn.Module):
def __init__(self, vocab, embed, hidden, layers):
super().__init__()
self.embedding = nn.Embedding(vocab, embed, padding_idx=0)
self.lstm = nn.LSTM(embed, hidden, num_layers=layers,
batch_first=True, dropout=0.3)
self.fc = nn.Linear(hidden, 1)
def forward(self, x):
out, (hn, cn) = self.lstm(self.embedding(x))
return torch.sigmoid(self.fc(hn[-1])).squeeze(1)Before each optimizer step: nn.utils.clip_grad_norm_(model.parameters(), 1.0) — the single most important line for stable RNN training.
Where RNNs Shine
Use a bidirectional RNN when the whole sequence is available (classification, tagging). Stick to unidirectional when you must generate or stream — you can't peek at the future.
RNN vs Transformer — What Changed
| Property | RNN / LSTM / GRU | Transformer |
|---|---|---|
| Parallelism | Sequential — step by step | Fully parallel |
| Long-range links | Must travel every step | Direct attention |
| Compute cost | O(T) — linear | O(T²) — quadratic |
| Streaming inference | Natural, one token at a time | Needs the full sequence |
| Small data | Often better | Needs lots of data |
Transformers dominate large-scale language, but RNNs remain the go-to for time-series, edge devices, streaming, and small-data problems. New hybrids like Mamba and RWKV blend both worlds.
7 Golden Rules for RNNs
A hidden state that carries context, weights shared across time, and gates to protect the gradient — that's the whole RNN family, from a 4-line cell to production LSTMs.