Multilayer Perceptron (MLP)

The Story That Explains an MLP

A Multilayer Perceptron (MLP) is the simplest fully connected feedforward neural network. Every neuron in one layer connects to every neuron in the next — no skips, no loops, no convolutions. Data flows in one direction: left to right, input to output. Despite that simplicity, MLPs can approximate any function — a fact so powerful it earned its own theorem.

Input, Hidden & Output Layers — The Three Tribes

Every MLP has exactly three kinds of layer. They have fundamentally different jobs, and confusing them is the fastest way to make wrong architectural decisions.

3D Architecture Diagram

The diagram below shows a 3 → 4 → 4 → 2 MLP: three input neurons, two hidden layers of four neurons each, and two output neurons. Every connection carries a trainable weight. Depth gives the network its hierarchical power.

Depth is not just about adding more neurons — it's about adding more representational stages. Here's how the same neuron budget changes in power depending on how it's arranged:

Hidden Layers & Depth — What Each Layer Learns

Depth is the primary lever for expressiveness in a neural network. Here is what that means in practice across common problem types:

Universal Approximation Theorem

Parameter Counting — The Exact Formula

Understanding parameter count is non-negotiable. It tells you memory footprint, training compute, overfitting risk, and whether your model can even fit on the hardware. The formula is always the same: weights + biases, layer by layer.

Python Code — MLP from Scratch to Training

Two implementations: scikit-learn (fast baseline) and PyTorch (transparent, production-ready). Both produce the same result on the same data.

from sklearn.neural_network import MLPClassifier
from sklearn.datasets import fetch_openml
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
from sklearn.metrics import accuracy_score

# Load MNIST (70,000 samples, 784 features)
X, y = fetch_openml('mnist_784', version=1, return_X_y=True, as_frame=False)
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42
)

# MLPs need feature scaling — pixel range 0-255 → 0-1
scaler = StandardScaler()
X_train = scaler.fit_transform(X_train)
X_test  = scaler.transform(X_test)

# Architecture: 784 → 256 → 128 → 10
mlp = MLPClassifier(
    hidden_layer_sizes=(256, 128),   # 2 hidden layers
    activation='relu',              # ReLU on every hidden neuron
    solver='adam',                  # adaptive moment optimiser
    learning_rate_init=1e-3,
    max_iter=30,
    batch_size=256,
    random_state=42,
    verbose=True
)

mlp.fit(X_train, y_train)
preds = mlp.predict(X_test)

print(f"Test Accuracy : {accuracy_score(y_test, preds):.4f}")
print(f"Total Params  : {sum(w.size for w in mlp.coefs_) + sum(b.size for b in mlp.intercepts_)}")

import torch
import torch.nn as nn
import torch.optim as optim
from torch.utils.data import DataLoader, TensorDataset

# ── Define the MLP ───────────────────────────────────────
class MLP(nn.Module):
    def __init__(self, input_dim, hidden_dims, output_dim):
        super().__init__()
        layers = []
        prev = input_dim
        for h in hidden_dims:
            layers.append(nn.Linear(prev, h))
            layers.append(nn.ReLU())
            layers.append(nn.BatchNorm1d(h))    # stabilises deep MLPs
            layers.append(nn.Dropout(0.3))       # regularisation
            prev = h
        layers.append(nn.Linear(prev, output_dim))
        self.net = nn.Sequential(*layers)

    def forward(self, x):
        return self.net(x)

# ── Instantiate: 784 → 256 → 128 → 10 ───────────────────
model = MLP(input_dim=784, hidden_dims=[256, 128], output_dim=10)

# ── Count parameters ─────────────────────────────────────
total_params    = sum(p.numel() for p in model.parameters())
trainable       = sum(p.numel() for p in model.parameters() if p.requires_grad)
print(f"Total params    : {total_params:,}")
print(f"Trainable params: {trainable:,}")

# ── Layer-by-layer breakdown ──────────────────────────────
for name, param in model.named_parameters():
    if param.requires_grad:
        print(f"  {name:30s} {str(param.shape):20s} {param.numel():>8,} params")

# ── Training loop (abbreviated) ──────────────────────────
optimizer = optim.Adam(model.parameters(), lr=1e-3)
criterion = nn.CrossEntropyLoss()

for epoch in range(1, 6):
    model.train()
    running_loss = 0.0
    for X_batch, y_batch in train_loader:
        optimizer.zero_grad()
        logits = model(X_batch)
        loss   = criterion(logits, y_batch)
        loss.backward()
        optimizer.step()
        running_loss += loss.item()
    print(f"Epoch {epoch} | Loss: {running_loss/len(train_loader):.4f}")

Golden Rules