Long Short-Term Memory (LSTM) Networks
Press Next → or use ← → arrow keys
The Detective Who Never Forgets
Think of the cell state as a filing room. Three guards control it: an archivist who shreds outdated files, an intake officer who screens new documents, and a spokesperson who decides what to reveal. Those are the three gates.
Why Vanilla RNNs Fail
During backpropagation through time, gradients are multiplied at every step. If the recurrent weights are small, that product collapses toward zero — distant clues become invisible to the optimiser.
Recurrent-weight eigenvalues below 1 → gradients vanish; above 1 → gradients explode into NaNs. LSTM's cell state sidesteps both with a constant error carousel.
LSTM Cell Anatomy
Each cell takes three things in and passes two things on. The cell state Cₜ is the long-term memory; the hidden state hₜ is the working output.
Unlike a vanilla RNN (which carries only hₜ), the LSTM threads both Cₜ and hₜ from step to step. That extra memory line is the whole advantage.
The Four Equations
Everything inside the cell is these six lines — three sigmoid gates, one tanh candidate, and two state updates:
| Step | Formula | Range |
|---|---|---|
| Forget gate | fₜ = σ(W_f·[hₜ₋₁, xₜ] + b_f) | (0, 1) |
| Input gate | iₜ = σ(W_i·[hₜ₋₁, xₜ] + b_i) | (0, 1) |
| Candidate | C̃ₜ = tanh(W_c·[hₜ₋₁, xₜ] + b_c) | (−1, 1) |
| Cell state | Cₜ = fₜ ⊙ Cₜ₋₁ + iₜ ⊙ C̃ₜ | — |
| Output gate | oₜ = σ(W_o·[hₜ₋₁, xₜ] + b_o) | (0, 1) |
| Hidden state | hₜ = oₜ ⊙ tanh(Cₜ) | (−1, 1) |
The cell state uses element-wise multiply (not matrix multiply) and an addition to combine old and new. That's precisely why gradients don't saturate away.
Gate Intuition — The Three Guards
Each gate is a security guard controlling the filing room. Together they decide what the cell remembers:
No one hand-writes the rules. The gates' weights are trained by backpropagation, so the network discovers for itself what to keep, add, and expose at each step.
LSTM Unrolled Through Time
Across time, each cell passes two lines forward — the cell state on top, the hidden state below — reusing the same weights at every step:
The green cell-state line carries long-term memory; the violet hidden line carries the short-term output. Both flow through every step, and every cell uses the same learned weights.
The LSTM Family Tree
The basic cell spawned a whole family, each tuned for a different job:
Every variant keeps the gated-memory idea — they just rearrange the connections for direction, depth, efficiency, or spatial structure.
Key Hyperparameters
These are the dials you actually turn when training an LSTM:
| Hyperparameter | Typical range | Tip |
|---|---|---|
| hidden_size | 64–512 | Start 128; double if underfitting |
| num_layers | 1–4 | 2 is usually enough; 3+ needs regularisation |
| dropout | 0.1–0.5 | 0.2–0.3 between layers (not recurrent) |
| sequence_length | ≤ 200 | Truncated BPTT beyond that |
| learning_rate | 1e-4–1e-2 | Adam 1e-3 with a scheduler |
| gradient_clipping | 0.5–5.0 | Always on — 1.0 is a safe default |
Begin with 1–2 layers of 128 units and Adam at 1e-3. Only add depth or width once you've confirmed the model is underfitting — deeper LSTMs overfit small datasets fast.
PyTorch — A Stock Forecaster
The task: predict tomorrow's closing price from the last 60 days. Two stacked LSTM layers feed a small dense head:
class LSTMForecaster(nn.Module):
def __init__(self):
super().__init__()
self.lstm = nn.LSTM(input_size=1, hidden_size=128,
num_layers=2, dropout=0.2, batch_first=True)
self.fc = nn.Sequential(
nn.Dropout(0.2), nn.Linear(128, 64),
nn.ReLU(), nn.Linear(64, 1))
def forward(self, x):
out, _ = self.lstm(x) # x: (batch, 60, 1)
return self.fc(out[:, -1, :]) # last step → priceTraining & Results
MSE loss, Adam at 1e-3, gradient clipping at 1.0, and a plateau scheduler over 50 epochs. The validation loss falls steadily:
The Same Model in Keras
Keras makes the stack even terser — just mind one rule about return_sequences:
import keras
from keras import layers
inputs = keras.Input(shape=(60, 1))
x = layers.LSTM(128, return_sequences=True, dropout=0.2)(inputs)
x = layers.LSTM(64, return_sequences=False, dropout=0.2)(x)
x = layers.Dense(32, activation='relu')(x)
outputs = layers.Dense(1)(x)
model = keras.Model(inputs, outputs)
model.compile(optimizer=keras.optimizers.Adam(1e-3, clipnorm=1.0),
loss='mse', metrics=['mae'])Stacked LSTMs must pass the full sequence forward; only the final LSTM returns just its last hidden state to feed the Dense head. Forget this and you get shape-mismatch errors.
LSTM vs Transformer
| Property | LSTM | Transformer |
|---|---|---|
| Memory | Recurrent cell state | Self-attention over all tokens |
| Long range | Good (60–200 steps) | Excellent (1000s) |
| Parallel training | Sequential | Fully parallel |
| Real-time inference | O(1) per step | Needs full context |
| Edge deployment | Lightweight | Often too large |
Transformers rule large-scale NLP, but for streaming ECG, financial ticks, sensor fusion, and on-device speech — where O(1) per-step inference and a small footprint matter — the LSTM is still the go-to.
Where LSTMs Live in Production
Wherever data arrives as an ongoing stream and decisions must be fast and local, the LSTM's constant-time step and small size make it the practical choice.
8 Golden Rules — Non-Negotiable
Clip, don't shuffle, normalise, reset state. Get these four right and you've avoided the vast majority of real-world LSTM failures.
The LSTM Cheat Sheet
iₜ = σ(W_i·[h,x]+b_i)
C̃ₜ = tanh(W_c·[h,x]+b_c)
Cₜ = fₜ⊙Cₜ₋₁ + iₜ⊙C̃ₜ
oₜ = σ(W_o·[h,x]+b_o)
hₜ = oₜ⊙tanh(Cₜ)
Binary → Dense(1), BCE, sigmoid
Multi-class → Dense(k), CE, softmax
Generation → Dense(vocab), CE, softmax
Gated memory, a family of variants, the hyperparameters that matter, and the code to ship — from a 60-day forecaster to a streaming medical monitor.