Optimizers in Deep Learning
Press Next → or use ← → arrow keys
The Blindfolded Hiker
Picture a hiker dropped somewhere on a foggy mountain range, blindfolded, trying to reach the lowest valley. They can only feel the slope under their feet. That slope is the gradient; the mountain is the loss landscape; and the walking strategy is the optimizer.
Every optimizer answers the same question — “given the slope, where do I step next?” — but the smart ones also remember momentum and adapt their stride to the terrain.
Gradient Descent — The Core Step
Every optimizer is a twist on one update: move the parameters a little bit in the direction that reduces the loss fastest — the negative gradient.
Too large and the hiker leaps over the valley; too small and they crawl. Nearly every practical failure of training traces back to a bad learning rate — not the choice of optimizer.
The Ravine Problem
Real loss surfaces have ravines — long narrow valleys, steep across, gently sloping along. Plain SGD overreacts to the steep walls and zig-zags, wasting most of its motion bouncing side to side instead of travelling down the valley floor.
The fix isn't a smaller learning rate (that just crawls) — it's memory. If the optimizer remembers its recent direction, side-to-side wobbles cancel out and forward progress adds up.
Momentum & Nesterov
Momentum gives the hiker inertia. Instead of stepping by the raw gradient, we accumulate a running velocity — a heavy ball rolling downhill smooths over bumps and powers through flat stretches.
Looks at the gradient where it is now, then adds it to the velocity. Damps oscillation and accelerates through ravines and plateaus.
First jumps ahead by the current velocity, then measures the gradient there. This peek forward corrects overshoot earlier — usually a little faster and more stable.
Along the valley floor gradients keep pointing the same way, so velocity compounds. Across the walls they flip sign every step, so they cancel. Momentum keeps the useful motion and kills the wasteful kind.
AdaGrad — A Rate Per Parameter
Momentum still uses one learning rate for every weight. But some features fire constantly and others are rare. AdaGrad gives each parameter its own learning rate, shrinking it in proportion to how much that parameter has already been updated.
Rare words and embeddings get large updates when they finally appear; common features get gently nudged. Ideal for bag-of-words and text models.
Gt only ever grows, so the effective learning rate marches monotonically toward zero. In a deep network, training eventually stalls and cannot recover.
RMSProp — Fading Memory
RMSProp keeps AdaGrad's per-parameter idea but fixes the dying rate. Instead of summing squared gradients forever, it keeps an exponentially-weighted moving average — recent terrain matters, ancient history fades.
The difference is dramatic. Watch the effective learning rate as training goes on:
Hinton introduced RMSProp in a 2012 Coursera class and never formally published it. It became the go-to for RNNs and reinforcement learning, where gradients swing wildly.
Adam — Best of Both Worlds
Adam (Adaptive Moment Estimation) fuses the two great ideas: momentum (a running average of the gradient) and RMSProp's per-parameter scaling (a running average of the squared gradient).
Momentum decides which way to go; the second moment decides how big each parameter's step should be. Together they converge fast on almost any architecture with little tuning.
Adam in Four Steps
The complete update per iteration, with defaults β1=0.9, β2=0.999, η=3e−4, ε=1e−8:
| Step | Formula | What it does |
|---|---|---|
| 1 — First moment | mt = β1mt−1 + (1−β1)∇L | running mean (momentum) |
| 2 — Second moment | vt = β2vt−1 + (1−β2)(∇L)2 | running mean of squares |
| 3 — Bias correction | m̂t = mt/(1−β1t) , v̂t = vt/(1−β2t) | fix the cold start |
| 4 — Update | θt+1 = θt − η · m̂t / (√v̂t + ε) | the actual step |
m and v both start at zero, so early estimates are biased toward zero — the optimizer would tiptoe for the first few steps. Dividing by (1−βt) rescales them so the very first step is already full-sized.
Bias Correction, Concretely
Take the very first step with β1=0.9 and a gradient g1=0.5. Without correction the momentum estimate is ten times too small:
The correction is huge exactly when it's needed (the start) and vanishes exactly when it isn't (later). No tuning, no warm-up hack — it just works.
Always Reach for AdamW
Plain Adam(weight_decay=...) mixes weight decay into the gradient, where the adaptive
scaling then distorts it. AdamW decouples weight decay — applying it directly to the weights —
which is what regularization was always supposed to do.
Decay is scaled by the per-parameter learning rate, so heavily-updated weights are barely regularized. The regularization strength becomes tangled with the gradient magnitude.
Weight decay is applied separately and uniformly. Cleaner generalization, and the reason it's the exclusive choice for training modern transformers.
GPT, BERT, T5 and essentially every large transformer are trained with
AdamW — typically lr=3e-4, weight_decay=0.01, plus a warm-up schedule.
Five Traps in the Landscape
Optimizers exist because the terrain fights back. These are the five hazards every training run must survive:
The Optimizer Scorecard
Four workhorses, side by side. There is no universal winner — only the right tool for the terrain.
| Property | SGD | AdaGrad | RMSProp | Adam |
|---|---|---|---|---|
| Adaptive LR | No | Yes | Yes | Yes |
| Momentum | Optional | No | Optional | Built-in |
| LR stability | Fixed | ↓ to 0 | Bounded | Bounded |
| Extra memory | 0 | 1 array | 1 array | 2 arrays |
| Bias correction | No | No | No | Yes |
| Best for | Vision | Sparse NLP | RNN / RL | Default |
| Generalization | Best | Good | Good | Sometimes worse |
Adam converges quickly but can land in sharp minima; well-tuned SGD+momentum finds flatter ones that transfer better — which is why vision leaderboards still favour it.
How They Converge
A typical training run: Adam plunges early, RMSProp tracks close behind, and SGD is slower but often settles to a lower final loss given enough epochs and a schedule.
A common recipe: start with Adam(3e-4) to get a model working fast, then, if every last point of accuracy matters, switch to tuned SGD+momentum with a learning-rate schedule.
What to Use, When
Matching the optimizer to the task saves far more time than tweaking it blindly:
| Task | Recommended setup | Why |
|---|---|---|
| Vision (CNNs) | SGD(lr=0.1, momentum=0.9) + CosineAnnealingLR | flatter minima, best generalization |
| NLP / Transformers | AdamW(lr=3e-4, weight_decay=0.01) + warmup | the field-wide standard |
| RNNs / LSTMs | RMSProp(lr=1e-3) + clip_grad_norm(1.0) | tames exploding gradients |
| Reinforcement learning | RMSProp(lr=2.5e-4) or Adam(3e-4) | adapts to non-stationary rewards |
| Sparse features | Adagrad(lr=0.05) or Adam | big updates for rare inputs |
optimizer = torch.optim.AdamW(model.parameters(),
lr=3e-4) — a strong, forgiving default for a first run on nearly any architecture.
Eight Golden Rules
optimizer.zero_grad() before loss.backward() — gradients accumulate by default.clip_grad_norm_(max_norm=1.0).The One-Screen Cheat Sheet
GD → add momentum (memory of direction) → add adaptivity (AdaGrad) → fix the dying rate (RMSProp) → combine both (Adam) → fix decay (AdamW).
Vision → SGD+mom. Transformers → AdamW. RNN/RL → RMSProp. Sparse → AdaGrad. Unsure → Adam(3e-4).
An optimizer turns a blindfolded stumble downhill into a confident, adaptive descent — remembering where it's been and adjusting its stride to the ground beneath each parameter.