PCA for Dimensional Reduction
Press Next → or use ← → arrow keys
The Best Angle To Cast A Shadow
PCA finds that angle. It rotates your coordinate system so the first new axis — the first principal component — points along the direction of maximum variance, the projection that keeps the most information when you flatten high dimensions down to a few.
PCA replaces your original axes with new ones ranked by how much the data spreads along them — so you can keep the top few and drop the rest, compressing the data while preserving nearly all of its structure.
Three Ideas That Make PCA Work
The high-variance directions are stable, reproducible structure; the near-flat directions are mostly measurement noise. Dimensional reduction is simply choosing to keep the former and discard the latter.
The Seven-Step PCA Recipe
B = X − μ, shifting the cloud's centre to the origin.C = BᵀB / (n−1).det(C − λI) = 0 — the variance of each PC.Z = B · W.Steps 4 and 5 are the whole story: the eigenvectors of the covariance matrix are the principal directions, and their eigenvalues are the variance along each. Everything else is setup and projection.
The Five Formulas
Stack the top-k eigenvectors into the weight matrix W and multiply the centred data by it. The result Z is your dataset re-expressed in the new, reduced coordinate system — same rows, fewer columns.
Four Points, By Hand — Setup
Take four samples with two features each and run the recipe end to end. First, centre the data on its mean.
With C in hand, the next step is det(C − λI) = 0 — two eigenvalues, two eigenvectors. The
diagram on the next slide shows exactly what those directions look like.
The Geometry — And The 1-D Projection
PC1 runs along the 45° diagonal (eigenvector [0.707, 0.707]). Projecting the four centred
points onto it gives a single coordinate each — the whole dataset compressed to one number per sample,
keeping 75% of the variance.
The Numbers That Fall Out
e₂ = [0.707, −0.707]
Run the same four points through PCA(n_components=1) and you get the identical projection to
machine precision. Working it by hand once removes all the mystery — sklearn is just doing this
eigen-decomposition for you, fast.
Eigenvectors Ask "Where?" · Eigenvalues Ask "How Much?"
That an eigenvalue equals the variance of the data projected onto its eigenvector isn't an arbitrary
label — it falls straight out of the eigenvector equation C·e = λ·e. The maths guarantees
the ranking is meaningful.
Variance Is Redistributed, Never Created
The eigenvalues always sum to the trace of the covariance matrix — the total variance of the original features. PCA doesn't add or destroy information, it redistributes it onto the new axes. Whatever variance a discarded component held is exactly the information you lose.
The Scree Plot & The Elbow
Plot the eigenvalues in descending order. The tall bars before the elbow are stable, reproducible signal; the flat tail is noise. Cut at the elbow — or wherever the cumulative curve crosses your variance target (commonly 95%).
What The Eigenvalues Are Telling You
| What You See | What It Means |
|---|---|
| First eigenvalue ≫ others (10×) | Data is effectively 1-D — check for a confounding variable |
| Eigenvalues nearly equal | Directions are numerically unstable & unreliable |
| High loadings on just 2 features | Those features explain most spread — a simpler model may match |
| Last few eigenvalues ≈ 0 | Safe to discard — unless hunting outliers/anomalies |
The eigenvectors of large eigenvalues barely move when new data arrives — they encode real, reproducible structure. Small-eigenvalue directions swing wildly with tiny perturbations. That's a second reason, beyond compression, to keep only the high-variance components: robustness.
What Dimensional Reduction Buys You
| Application | Why PCA Helps | When To Use |
|---|---|---|
| Visualization | Collapse to 2-D/3-D for scatter plots | Always 2–3 PCs for plotting |
| Noise filtering | Drop low-eigenvalue components | Strip measurement noise |
| Multicollinearity fix | Build uncorrelated features | Before OLS / regression (PCR) |
| Training speed-up | 500 features → 50 PCs | Accelerate SVM / kNN |
| Feature extraction | Surface latent patterns | Discover hidden structure |
Tree models (Random Forest, XGBoost) already handle high dimensions natively — PCA rarely helps them. Skip it too when the structure is non-linear (reach for kernel PCA, t-SNE or UMAP) or when you need interpretable features (use feature selection instead).
By Hand, Then Verified With sklearn
import numpy as np from sklearn.decomposition import PCA X = np.array([[1,2], [3,4], [5,4], [3,6]], dtype=float) # ── manual steps ── mu = X.mean(axis=0) # [3. 4.] B = X - mu # mean-centre C = np.cov(B, rowvar=False) # covariance matrix vals, vecs = np.linalg.eigh(C) # eigenvalues + eigenvectors # ── sklearn verification ── Z = PCA(n_components=1).fit_transform(X) print(Z.ravel()) # ≈ [-2.83, 0, 1.41, 1.41] — matches the hand maths
Confirm sum(eigenvalues) == trace(C) — total variance is conserved. If it doesn't hold,
you've made an arithmetic slip somewhere. It's the fastest way to catch a broken PCA by hand.
PCA vs Other Reduction Methods
| Property | PCA | t-SNE | UMAP | LDA |
|---|---|---|---|---|
| Linearity | Linear | Non-linear | Non-linear | Linear |
| Supervision | Unsupervised | Unsupervised | Unsupervised | Supervised |
| Preserves | Global variance | Local clusters | Local + global | Class separability |
| ML-pipeline safe? | Yes | No (stochastic) | Sometimes | Yes |
| Best for | Preprocessing | 2-D/3-D viz | Exploration | Classification prep |
Use PCA as your default preprocessing and denoising step — it's linear, fast, and pipeline-safe. Switch to t-SNE / UMAP purely for visual cluster exploration, and to LDA when the goal is maximizing class separation before a classifier.
Seven Non-Negotiables
You've Seen PCA All The Way Down
Dimensional reduction with PCA is one clean idea: mean-centre, find the covariance's eigenvectors, keep the directions of greatest variance, and project. The eigenvalues tell you exactly how much you keep and how much you lose — total variance is conserved, never conjured.
Re-run the four-point example yourself, then scale up to Iris and the digits dataset. Compare TruncatedSVD on large sparse data, and explore kernel PCA and UMAP when your data refuses to be linear.
🗜️ End of tutorial · Press ← to review, or click Restart