Linear Discriminant Analysis
Press Next → or use ← → arrow keys
The Intuition — The Museum Guard
That question is LDA. It searches for the projection that maximizes the gap between classes while minimizing the spread within each class — turning a messy multi-feature problem into one clean, separating axis.
Invented by Ronald Fisher in 1936, LDA uses the class labels PCA ignores. Where PCA keeps the directions of greatest total variance, LDA keeps the directions of greatest class separability — a crucial difference when your end goal is classification.
LDA Wears Two Hats
Whether you want a compact 2-D view of a 100-feature dataset or a quick, accurate classifier, LDA solves the same optimization — it just either hands you the projected coordinates or the predicted labels.
Maximize Between · Minimize Within
Between-class scatter (Sᵦ) measures how far the class centroids sit from each other — push it up. Within-class scatter (Sᵥᵥ) measures how spread out each class is — push it down. LDA finds the direction that maximizes their ratio.
The Right Axis Untangles The Classes
Project the two classes onto the wrong axis and they smear together — no threshold can split them. Project onto the LDA axis and the blues land on one side, the ambers on the other. Finding that axis is the entire job of LDA.
The Fisher Criterion
Maximizing the Fisher ratio turns, through calculus, into the generalized eigenvalue problem
Sᵥᵥ⁻¹Sᵦ w = λw. Each eigenvalue tells you how much class separation its direction
captures — exactly analogous to PCA's eigenvalues measuring variance.
LDA In Six Steps
Sᵥᵥ⁻¹Sᵦ w = λw for eigenvalues and eigenvectors.Z = X · W.Because Sᵦ is built from only C class means, it has rank at most C−1 — so LDA can produce at most C−1 discriminant axes, no matter how many input features you have. A 3-class problem yields at most 2; a 2-class problem, just 1.
Two Wines, By Hand
Four wine samples, two features (Alcohol %, Malic Acid), two classes — Barolo vs Barbera. What single direction best tells them apart?
μ₂ = [12.15, 3.3]
[−0.14, 0.1]]
[−2.77, 2.25]]
With a large Sᵦ and a small Sᵥᵥ, the Fisher ratio is high along w. Projecting all four
samples onto that single direction lines the Barolos up on one side and the Barberas on the other —
two features reduced to one perfectly separating score.
PCA Chases Variance · LDA Chases Separation
Here the greatest spread runs along the diagonal both classes share — so PCA's top axis barely separates them. LDA instead looks across the classes and finds the direction that pulls blue and amber apart. When your goal is classification, variance is the wrong compass.
PCA vs LDA — Side By Side
| Property | PCA | LDA |
|---|---|---|
| Type | Unsupervised | Supervised |
| Objective | Maximize total variance | Maximize class separability |
| Uses labels? | No | Yes |
| Max components | min(n−1, d) | C − 1 |
| Gaussian assumption | No | Yes (per class) |
| Best for | Viz · noise removal · unlabeled | Classification preprocessing |
The pros use both: run PCA first to strip noise and fix singularity when features outnumber samples, then run LDA on the cleaned-up data to extract the discriminant axes. That two-stage pipeline is exactly what powers Fisherfaces.
The Fine Print — Four Assumptions
| Assumption | What It Means | If Violated |
|---|---|---|
| Gaussian classes | Each class is multivariate normal | Transform features (log, Box-Cox) |
| Equal covariance | Classes share one covariance shape | Switch to QDA |
| Independent samples | No time-series / repeated measures | Use mixed / multilevel models |
| Non-singular Sᵥᵥ | Needs n > d to invert Sᵥᵥ | PCA first, or shrinkage |
LDA tolerates gentle departures from normality, so don't panic over slightly skewed features. The two that really bite are unequal covariances (fix with QDA) and the singularity problem when d ≥ n (fix with PCA-first or shrinkage).
When Equal-Covariance Fails → QDA
| Aspect | LDA | QDA |
|---|---|---|
| Decision boundary | Linear hyperplane | Quadratic (curved) surface |
| Covariance | One shared Σ | One Σ per class |
| Bias | Higher (if Σ differ) | Lower |
| Variance | Lower (fewer params) | Higher (more params) |
| Needs more data? | No | Yes |
| Dimensionality reduction | Yes (C−1 axes) | No |
LDA fits one shared covariance — fewer parameters, lower variance, but biased when the classes truly differ in shape. QDA fits a covariance per class — flexible curved boundaries, but hungry for data. If a Box's M test flags unequal covariances and you have the samples, QDA usually wins.
Regularized LDA & Choosing A Solver
When samples are scarce, Sᵥᵥ is estimated badly. Regularized LDA blends it toward a
simple diagonal target — Sᵥᵥ(λ) = (1−λ)·Sᵥᵥ + λ·(tr Sᵥᵥ/d)·I — where λ=0 is standard LDA
and λ=1 is fully shrunk. scikit-learn picks λ for you with shrinkage='auto' (Ledoit-Wolf).
| Solver | Shrinkage? | predict_proba? | Best Use |
|---|---|---|---|
svd (default) | No | Yes | Standard LDA, n > d |
lsqr | Yes | No | High-dim, small sample |
eigen | Yes | Yes | Probabilities + regularization |
LDA As Reducer And As Classifier
from sklearn.discriminant_analysis import LinearDiscriminantAnalysis from sklearn.preprocessing import StandardScaler from sklearn.pipeline import Pipeline # ── as a dimensionality reducer ── X_scaled = StandardScaler().fit_transform(X) lda = LinearDiscriminantAnalysis(n_components=2) X_lda = lda.fit_transform(X_scaled, y) # needs labels y! print(lda.explained_variance_ratio_) # Wine: [0.688, 0.312] # ── as a classifier (baseline) ── pipe = Pipeline([('sc', StandardScaler()), ('lda', LinearDiscriminantAnalysis())]) pipe.fit(X_train, y_train) pipe.predict(X_test) # Wine CV acc ≈ 98.9%
On roughly-Gaussian tabular data LDA is astonishingly strong for how little it costs — a matrix multiply and a nearest-centroid check. On the Wine dataset it hits ~98.9% cross-validated accuracy out of the box. Beat that before reaching for anything heavier.
PCA → LDA — The Fisherfaces Pipeline
You can't run LDA on 4096 features with only 400 faces — Sᵥᵥ is singular. PCA first squeezes the images to 150 denoised dimensions, then LDA extracts the 39 most discriminative directions. The combo lifts accuracy from 78.5% → 93.2% — a +14.7% jump from adding the LDA stage.
Where LDA Shines
LDA's discriminant coefficients double as feature importances. On breast-cancer data the top drivers — worst concave points, worst perimeter, mean concave points — line up with known malignancy markers, so the model's reasoning is easy to defend.
When To Use LDA — And When Not To
| ✅ Reach For LDA When… | ❌ Avoid LDA When… |
|---|---|
| Multi-class tabular data with roughly-Gaussian features | Features are binary, count, or heavily skewed |
| You want to reduce to C−1 dims before SVM / kNN / NN | Classes are very imbalanced (set priors, or use another model) |
| Face / biometric recognition (PCA → LDA) | Boundaries are non-linear (spiral, ring, XOR) |
| You need a fast, interpretable baseline classifier | Data is unlabeled — use PCA instead |
LDA draws straight decision boundaries. For genuinely curved class structure, reach for kernel LDA, an SVM with an RBF kernel, or a neural network. And for heavily non-Gaussian features, tree models, Naive Bayes, or logistic regression usually serve better.
Seven Rules For Using LDA Well
predict(). It's a fast, strong baseline.You Now Understand LDA End-To-End
LDA uses the labels PCA ignores to find the projection that maximizes between-class separation over within-class spread — the Fisher criterion. It reduces to at most C−1 axes, classifies by nearest centroid, and pairs beautifully with PCA for high-dimensional problems like face recognition.
Try LDA on Iris, Wine and Breast-Cancer, compare it against QDA when covariances differ, build a Fisherfaces pipeline on the Olivetti faces, and reach for kernel LDA when the boundary curves.
🎯 End of tutorial · Press ← to review, or click Restart