LSTM Networks — From Cell State to Gates
Press Next → or use ← → arrow keys
What Are LSTM Networks?
An LSTM carries a cell state (long-term storage) alongside the hidden state (short-term working memory). Keeping them separate is the whole trick.
Why We Need LSTMs
A vanilla RNN's memory fades fast. Each step multiplies the signal by a weight below 1, so information decays exponentially — often gone within seven steps:
To fill the blank you must still remember "clouds" and "sky" — but a vanilla RNN has lost them by the time it reaches the gap. The LSTM was built to solve exactly this.
The Cell State — A Conveyor Belt
Picture a conveyor belt running straight through the whole network. Packages of information ride along it, barely touched — this is the cell state, the LSTM's protected long-term memory.
The cell state itself changes very little from step to step. Small, learned gates decide which packages board, ride on, or get thrown off — leaving the rest untouched.
Inside the LSTM Cell
The cell state runs across the top. Below it, the inputs [hₜ₋₁, xₜ] drive three gates that edit the belt and read from it:
The forget gate multiplies the belt (erase), the input gate adds new values (write), and the output gate reads a filtered copy out as the hidden state hₜ.
Gate 1 — The Forget Gate
First, decide what to erase from the old cell state. A sigmoid outputs a keep-fraction between 0 and 1 for every value on the belt:
While reading "are very", the forget gate stays near 1.0 for "clouds" and "sky" — protecting them — while letting irrelevant earlier words fade.
Gate 2 — Input Gate & Candidate
Next, decide what new information to write. Two parts work together: a candidate proposes values, and the input gate controls how much of it gets through.
The product iₜ ⊙ C̃ₜ is the scaled new information. On "clouds" the gate opens wide (iₜ ≈ 0.9); on a filler word like "in" it nearly shuts (iₜ ≈ 0.05), so the belt barely changes.
The Cell State Update — The Core
This one equation is the heart of the LSTM. Erase with the forget gate, add the new writing — then move on:
Nothing here squashes the whole memory through a dense layer. The cell state flows forward with only a scale-and-add — a nearly uninterrupted path from the distant past to now.
Gate 3 — The Output Gate
Finally, decide what to reveal. The protected cell state is squashed by tanh, then filtered by the output gate to form this step's hidden state:
An LSTM can hold information without exposing it. The output gate stays low until the moment it matters — in our sentence it opens fully (oₜ ≈ 1.0) only when predicting the final word.
Why Addition Beats Multiplication
In a vanilla RNN the gradient is multiplied by a weight at every step, so it shrinks exponentially. The LSTM's cell state uses addition, and the gradient through it stays bounded:
Because the cell state carries the gradient by a scale-and-add rather than repeated matrix multiplication, the error signal rides all the way back to early steps — undamaged.
The Complete Equations
Three gates, one candidate, and two state updates — the whole LSTM on one screen:
| Component | Formula | Range |
|---|---|---|
| Forget gate | fₜ = σ(W_f·[hₜ₋₁, xₜ] + b_f) | (0, 1) |
| Input gate | iₜ = σ(W_i·[hₜ₋₁, xₜ] + b_i) | (0, 1) |
| Candidate | C̃ₜ = tanh(W_c·[hₜ₋₁, xₜ] + b_c) | (−1, 1) |
| Cell update | Cₜ = fₜ·Cₜ₋₁ + iₜ·C̃ₜ | — |
| Output gate | oₜ = σ(W_o·[hₜ₋₁, xₜ] + b_o) | (0, 1) |
| Hidden state | hₜ = oₜ · tanh(Cₜ) | (−1, 1) |
W_f, W_i, W_c, W_o are shared across all time steps and learned by backpropagation through time — the parameter count never grows with sequence length.
Worked Example — "The Clouds in the Sky Are Very ___"
Trace the gates as the sentence streams in, word by word:
A vanilla RNN would have lost "clouds" by the blank. The LSTM's belt carried it through untouched — forget ≈ 1 for what mattered, input ≈ 0 for what didn't.
Cell State vs Hidden State
The two memories play very different roles — decoupling them by design is what makes LSTMs work:
| Cell state Cₜ | Hidden state hₜ | |
|---|---|---|
| Role | Long-term memory | Short-term working memory |
| Access | Private, protected | Public — passed to next layer |
| Path | Scale + add (the highway) | Squashed & gated each step |
| Updated by | Forget + input gates | Output gate · tanh(Cₜ) |
The cell state hoards information safely; the hidden state exposes only the slice that's useful right now. Separating them is the design insight the whole architecture rests on.
LSTM in PyTorch
The four equations map straight onto four linear layers — or you can just call the fused, GPU-optimised built-in:
# A single LSTM cell, by hand
combined = torch.cat([h_prev, x], dim=1)
f = torch.sigmoid(self.Wf(combined)) # forget gate
i = torch.sigmoid(self.Wi(combined)) # input gate
c_tilde = torch.tanh(self.Wc(combined)) # candidate
c = f * c_prev + i * c_tilde # cell update
o = torch.sigmoid(self.Wo(combined)) # output gate
h = o * torch.tanh(c) # hidden state
# ...or the built-in (four matrices fused for speed)
model = nn.LSTM(input_size=10, hidden_size=128, num_layers=2,
dropout=0.2, batch_first=True)
out, (hn, cn) = model(x) # out: (batch, seq, 128)nn.LSTM concatenates the four weight matrices into one fused operation — mathematically identical to the hand-written cell, but far quicker on a GPU.