LSTM Networks From Cell State to Gates

What Are LSTM Networks?

Long Short-Term Memory networks — usually just called LSTMs — are a special kind of Recurrent Neural Network (RNN), capable of learning long-term dependencies. They were introduced by Hochreiter & Schmidhuber (1997) and were refined and popularized by many researchers in the work that followed. They work tremendously well on a large variety of problems and are now widely used across industry and research.

🖱️ Click the diagram to pause. The glowing middle cell (t) is the currently active one. The blue horizontal line is the cell state — the long-term memory highway running unchanged through the chain.

Why Do We Need LSTMs?

Traditional RNNs often struggle to retain information over long sequences due to issues like vanishing and exploding gradients.

How Do LSTMs Work? — The Cell State Conveyor Belt

The crucial innovation in LSTMs is the cell state. Think of it as a conveyor belt that carries information through the network. Information can flow along this conveyor belt almost unchanged, allowing long-term information retention.

The key to LSTMs is the cell state — the horizontal line running through the top of the diagram. The cell state runs straight down the entire chain, with only some minor linear interactions. It's very easy for information to just flow along it unchanged.

The packages on the conveyor belt represent pieces of information ("clouds", "sky", "plural") being carried unchanged across time steps. Gates decide what to load, remove, or read.

The Complete LSTM Cell — Step-by-Step Interactive

An LSTM cell consists of three gates and one candidate state computation. The interactive diagram below lets you click each gate to see what it does — each step lights up in sequence. Use the buttons to walk through each stage.

Gate 1 — The Forget Gate

Gate 2 — The Input Gate & Candidate Values

The Cell State Update — Cₜ = fₜ · Cₜ₋₁ + iₜ · C̃ₜ

The overall cell state update equation is the heart of the LSTM. This single equation combines both gates to produce the new memory state.

Gate 3 — The Output Gate

Complete Equation Summary

Worked Example — "The Clouds in the Sky Are Very ____"

Let's trace an LSTM through the sentence word-by-word to see exactly what each gate does.

Real-World Applications of LSTMs

Python Implementation — PyTorch LSTM in Practice

Below is a clean, fully annotated PyTorch LSTM that mirrors the exact architecture we've studied — forget, input, candidate, and output gates under the hood.

import torch
import torch.nn as nn

class LSTMCell_Manual(nn.Module):
    """
    Manual LSTM cell — implements all 4 equations explicitly.
    Matches the PPT diagram step by step.
    """
    def __init__(self, input_size, hidden_size):
        super().__init__()
        n = input_size + hidden_size

        # Four weight matrices (one per gate/candidate)
        self.Wf = nn.Linear(n, hidden_size)  # Forget gate   → fₜ
        self.Wi = nn.Linear(n, hidden_size)  # Input gate    → iₜ
        self.Wc = nn.Linear(n, hidden_size)  # Candidate     → C̃ₜ
        self.Wo = nn.Linear(n, hidden_size)  # Output gate   → oₜ

    def forward(self, x, h_prev, c_prev):
        # Concatenate previous hidden state + current input
        combined = torch.cat([h_prev, x], dim=1)  # [hₜ₋₁, xₜ]

        # ── Step 1: Forget Gate ──────────────────────────────
        f_t = torch.sigmoid(self.Wf(combined))
        # f_t ∈ [0,1] — how much of old cell state to keep

        # ── Step 2: Input Gate + Candidate ───────────────────
        i_t   = torch.sigmoid(self.Wi(combined))
        c_tld = torch.tanh   (self.Wc(combined))
        # i_t   ∈ [0,1]  — how much new info to write
        # c_tld ∈ [-1,1] — candidate values to potentially write

        # ── Step 3: Cell State Update ────────────────────────
        c_t = f_t * c_prev + i_t * c_tld
        # Cₜ = fₜ ⊙ Cₜ₋₁  +  iₜ ⊙ C̃ₜ
        # ADDITION = gradient highway (no vanishing!)

        # ── Step 4: Output Gate ──────────────────────────────
        o_t = torch.sigmoid(self.Wo(combined))
        h_t = o_t * torch.tanh(c_t)
        # hₜ = oₜ ⊙ tanh(Cₜ)

        return h_t, c_t  # new hidden state, new cell state


# ── Using PyTorch's built-in (optimised CUDA version) ─────
model = nn.LSTM(
    input_size  = 10,    # xₜ dimension
    hidden_size = 128,   # hₜ and Cₜ dimension
    num_layers  = 2,     # stacked LSTMs
    dropout     = 0.2,   # between layers only
    batch_first = True,  # input: (batch, seq_len, features)
    bidirectional = False
)

# Forward pass
x   = torch.randn(4, 20, 10)    # batch=4, seq=20, features=10
h0  = torch.zeros(2, 4, 128)    # (num_layers, batch, hidden)
c0  = torch.zeros(2, 4, 128)

output, (hn, cn) = model(x, (h0, c0))
print(f"Output shape: {output.shape}")     # (4, 20, 128)
print(f"Final hidden: {hn.shape}")          # (2, 4, 128)
print(f"Final cell:   {cn.shape}")          # (2, 4, 128)

Key Takeaways — What You Now Know