Activation Functions in Deep Learning
Press Next → or use ← → arrow keys
Why Neurons Must Be Non-Linear
A good activation must (1) be non-linear so depth adds power, and (2) be differentiable almost everywhere so gradients can flow backward during training.
Sigmoid — The Classic Squash
The sigmoid squeezes any input into a smooth (0, 1) range — perfect for probabilities. It ruled early neural nets, but it hides a costly flaw.
| Formula | σ(z) = 1 / (1 + e⁻ᶻ) |
| Range | (0, 1) |
| Max gradient | σ′(z) ≤ 0.25 |
| Zero-centred? | No |
Its output is always positive, so gradient updates zig-zag. And its slope never exceeds 0.25 — so in a deep stack, gradients shrink fast. Today sigmoid survives only on binary output neurons.
tanh — Sigmoid, Recentred
Tanh has the same S-shape but spans (−1, +1), so its outputs are zero-centred — gradients flow more cleanly than sigmoid. Its slope can reach 1.0, four times sigmoid's.
| Formula | (eᶻ − e⁻ᶻ)/(eᶻ + e⁻ᶻ) |
| Range | (−1, +1) |
| Max gradient | tanh′(z) ≤ 1.0 |
| Zero-centred? | Yes |
Tanh still saturates: push |z| large and the curve flattens, so the gradient still dies at the extremes. The saturation trap is next.
The Saturation Trap
Each layer can multiply the gradient by ≤ 0.25. After 10 layers that's 0.25¹⁰ ≈ 0.000001 — the early layers effectively stop learning. This single problem is why ReLU took over.
ReLU — The Revolution
ReLU is almost embarrassingly simple: keep positives, zero the rest. Yet it trains up to 6× faster than sigmoid and sparked the 2010s deep-learning boom.
| Formula | ReLU(z) = max(0, z) |
| Range | [0, ∞) |
| Gradient (z > 0) | exactly 1 — no vanishing |
| Gradient (z < 0) | 0 — dead zone |
For positive inputs the gradient is exactly 1, so it passes backward untouched — no saturation, no shrinkage. It's also just a comparison, so it's cheap to compute at scale.
ReLU's Fatal Flaw — Dying Neurons
Aggressive learning rates or poor initialisation can kill nearly half a layer. The fixes all share one idea: let a little signal leak through on the negative side.
Leaky ReLU & ELU — Reviving the Dead
Both keep ReLU's positive side but replace the flat zero with a gentle negative slope, so neurons keep a pulse.
A tiny slope (α ≈ 0.01) on the negatives means gradients never fully vanish. Cheap and effective — a favourite for CNNs and GANs.
A smooth exponential curve gives near-zero-centred outputs and a clean gradient everywhere — great for deep MLPs, at a small compute cost.
GELU — The Transformer's Choice
GELU multiplies each input by the probability it's positive under a Gaussian: f(z) = z · Φ(z). The result is a smooth curve that dips slightly below zero before rising — a soft, probabilistic gate.
| Formula | z · Φ(z) |
| Min value | ≈ −0.17 near z = −0.75 |
| Smooth? | Yes — everywhere |
| Used in | BERT, GPT-2/3/4 |
Its smooth, near-zero-centred shape produces cleaner gradient flow through residual connections and attention than ReLU's hard kink — which is why every major LLM uses it by default.
The ReLU Family — Side by Side
| Activation | Dying neurons | Zero-centred | Smooth | Cost | Best for |
|---|---|---|---|---|---|
| ReLU | Yes — common | No | No (kink) | Very low | CNNs, fast baseline |
| Leaky ReLU | No | No | No | Very low | CNNs, GANs |
| ELU | No | Yes (≈) | Yes | Medium | Deep MLPs, regression |
| GELU | No | Yes (≈) | Yes | Medium | Transformers, LLMs |
As you move down the table you trade a little compute for smoother, zero-centred, always-alive neurons. For anything deeper than a few layers, that trade is almost always worth it.
Softmax — Turning Scores into Probabilities
Softmax takes a vector of raw scores (logits) and turns them into probabilities that sum to 1 — the standard final layer for multi-class classification.
softmax(z)ᵢ = eᶻⁱ / Σⱼ eᶻʲ forces neurons into a zero-sum competition. In hidden layers that destroys the independent signal each neuron carries — so it belongs only in the output.
All Activations From Scratch
Every activation is a one-liner in NumPy:
import numpy as np
def sigmoid(z): return 1 / (1 + np.exp(-z))
def tanh(z): return np.tanh(z)
def relu(z): return np.maximum(0, z)
def leaky_relu(z, alpha=0.01): return np.where(z > 0, z, alpha * z)
def elu(z, alpha=1.0): return np.where(z > 0, z, alpha * (np.exp(z) - 1))
def gelu(z):
return 0.5 * z * (1 + np.tanh(np.sqrt(2/np.pi) * (z + 0.044715 * z**3)))
def softmax(z):
e = np.exp(z - np.max(z)) # subtract max for stability
return e / e.sum()
print(softmax(np.array([3.2, 1.8, 0.5]))) # [0.706 0.176 0.118]Subtracting max(z) before exponentiating prevents overflow on large logits without changing the result — always do it.
Activations in a Real Network
A modern deep MLP: GELU in the hidden layers, LayerNorm between them, and raw logits at the output.
import torch.nn as nn
class DeepMLP(nn.Module):
def __init__(self, in_dim, hidden, out_dim):
super().__init__()
self.net = nn.Sequential(
nn.Linear(in_dim, hidden),
nn.GELU(), # ← modern choice
nn.LayerNorm(hidden),
nn.Linear(hidden, hidden),
nn.GELU(),
nn.LayerNorm(hidden),
nn.Linear(hidden, out_dim), # raw logits — no softmax here
)
def forward(self, x): return self.net(x)nn.CrossEntropyLoss already applies log-softmax internally. Adding your own nn.Softmax() applies it twice — corrupting gradients and causing numerical instability.
How to Choose — The Decision Map
| Situation | Use this |
|---|---|
| Output layer — multi-class | Softmax |
| Output layer — binary | Sigmoid |
| Transformers / LLMs | GELU |
| CNNs / vision | ReLU → Leaky ReLU if >10% die |
| Deep MLP / tabular | ELU or GELU |
| GAN discriminator | Leaky ReLU (α = 0.2) |
Use GELU in hidden layers and nn.CrossEntropyLoss (which handles softmax) for classification. It's the setup behind every major transformer and almost always beats ReLU beyond four layers.
6 Golden Rules
From sigmoid to GELU, the whole story is about keeping gradients alive as networks grow deep. Choose the curve well and everything downstream trains faster and further.