The Multilayer Perceptron
Press Next → or use ← → arrow keys
What Is an MLP, Really?
Information flows in one direction — input → hidden → output — with no loops. That single, tidy assumption is what makes an MLP so easy to train and so widely used as the first neural network anyone learns.
The Three Kinds of Layers
Every MLP is built from three roles. Get these straight and the rest of the network makes sense.
A single perceptron can only draw a straight line. Stack them with nonlinear activations and you get a multilayer perceptron — able to carve curved, tangled decision boundaries no single line could.
The Forward Pass, Step by Step
One neuron does two things: a weighted sum, then a nonlinear squash. Repeat layer by layer and the input becomes a prediction.
| Stage | Computation | What comes out |
|---|---|---|
| Input | x = [x₁, x₂, x₃] | Raw feature vector |
| Hidden 1 | a₁ = ReLU(W₁·x + b₁) | 4 activations |
| Hidden 2 | a₂ = ReLU(W₂·a₁ + b₂) | 4 activations |
| Output | ŷ = softmax(W₃·a₂ + b₃) | 2 class probabilities |
Remove the activation function and a hundred stacked layers collapse into a single linear one. ReLU, sigmoid or tanh are what let depth actually buy you expressive power.
Anatomy of a 3 → 4 → 4 → 2 Network
Here is a complete MLP: three inputs, two hidden layers of four neurons each, and two outputs — every neuron wired to every neuron in the next layer.
People describe an MLP by its layer sizes: [3, 4, 4, 2]. The first number is features in, the last is classes out, and everything between is your design space.
Shallow vs Deep: Why Depth Matters
A shallow network is one detective trying to crack the whole case alone. A deep network is a relay of detectives — each hands a partial insight to the next, so the final one solves what none could solve solo. Depth is teamwork across layers.
What Each Layer Learns
In a deep network, features grow in abstraction layer by layer. For a face-recognition net, the progression is strikingly intuitive:
How much depth you need depends on the domain:
| Domain | Typical depth | Why |
|---|---|---|
| Tabular / simple | 1–3 layers | Few interactions to model |
| Images | Tens of layers | Deep visual hierarchy |
| Language | Dozens+ layers | Long-range, abstract structure |
The Catch: Vanishing Gradients
Depth is powerful, but naive deep networks are hard to train. During backpropagation, the error signal is multiplied layer by layer on its way back — and if those multipliers are small, it shrinks toward zero.
By the time the signal reaches the first layers, it's a whisper — so those layers barely update. The modern fixes: ReLU activations, batch normalization, careful weight initialization, and residual connections.
The Universal Approximation Theorem
The theorem guarantees a network exists — it says nothing about whether gradient descent will find it, how much data you'll need, or how wide the layer must be. In practice, depth reaches the same functions far more efficiently than a single monstrously-wide layer.
Counting the Parameters
Every connection is a weight, and every neuron (except inputs) has a bias. For a layer with n inputs and m neurons: (n × m) weights + m biases.
| Layer | Weights (n × m) | Biases | Subtotal |
|---|---|---|---|
| Input → Hidden 1 | 3 × 4 = 12 | 4 | 16 |
| Hidden 1 → Hidden 2 | 4 × 4 = 16 | 4 | 20 |
| Hidden 2 → Output | 4 × 2 = 8 | 2 | 10 |
| Total | 3→4→4→2 network | 46 | |
Width is expensive — parameters grow with the product of adjacent layer sizes:
| Architecture | ≈ Parameters |
|---|---|
| 784 → 128 → 10 | ~101,770 |
| 784 → 512 → 256 → 10 | ~535,818 |
| 784 → 1024 → 1024 → 10 | ~1.87 million |
Doubling a layer's width roughly doubles the weights on both sides of it. More parameters means more memory, slower training, and a bigger appetite for data before overfitting sets in.
An MLP in scikit-learn
For tabular data and quick baselines, MLPClassifier gives you a working network in a handful of lines — no gradients to hand-code.
from sklearn.neural_network import MLPClassifier
from sklearn.preprocessing import StandardScaler
from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2)
# always scale inputs before a neural net
scaler = StandardScaler().fit(X_train)
X_train, X_test = scaler.transform(X_train), scaler.transform(X_test)
clf = MLPClassifier(
hidden_layer_sizes=(4, 4), # two hidden layers of 4
activation="relu",
solver="adam",
max_iter=500,
early_stopping=True,
)
clf.fit(X_train, y_train)
print(clf.score(X_test, y_test))Neural nets are sensitive to feature magnitude. Standardizing inputs to mean 0 / variance 1 keeps gradients well-behaved and training stable — skip it and the net may never converge.
The Same MLP in PyTorch
When you need full control — custom layers, GPUs, your own training loop — define the network as an nn.Module.
import torch.nn as nn
class MLP(nn.Module):
def __init__(self, in_dim, n_classes):
super().__init__()
self.net = nn.Sequential(
nn.Linear(in_dim, 64),
nn.BatchNorm1d(64),
nn.ReLU(),
nn.Dropout(0.3),
nn.Linear(64, 32),
nn.ReLU(),
nn.Linear(32, n_classes),
)
def forward(self, x):
return self.net(x) # logits; softmax lives in the loss
model = MLP(in_dim=20, n_classes=3)
loss_fn = nn.CrossEntropyLoss()
opt = torch.optim.Adam(model.parameters(), lr=1e-3)The model outputs raw logits; CrossEntropyLoss applies softmax internally for numerical stability. Only add an explicit softmax at inference time when you want probabilities.
Three Tools That Make Deep MLPs Work
The difference between an MLP that trains and one that stalls or overfits usually comes down to these three:
A common, reliable block is Linear → BatchNorm → ReLU → Dropout. Normalize before the nonlinearity, drop out after it.
7 Golden Rules for Building MLPs
The MLP is the foundation every fancier architecture builds on. Master its layers, forward pass, and training tricks, and CNNs, RNNs and Transformers become variations on a theme you already understand.