Principal Component Analysis
Press Next → or use ← → arrow keys
The Intuition — Find The Best Shadow
PCA finds that best torch angle. It looks for the directions along which your data is most spread out, and projects the high-dimensional cloud down onto them — keeping the shape that matters and throwing away the flat, uninformative directions.
PCA rotates your data into a new frame whose axes point along the directions of maximum variance, ordered largest-first — so you can drop the last few axes and lose almost no information.
Why Reduce Dimensions At All?
Feature selection keeps a subset of your original columns. PCA is different: it builds brand-new columns — each a linear combination of all the originals — chosen to capture as much spread as possible in as few axes as possible.
Projecting Onto The Principal Axes
PC1 points along the direction the cloud stretches most. PC2 is perpendicular to it and captures the next-most spread. Dropping to PC1 alone projects every point onto that line — reducing 2-D to 1-D while keeping the bulk of the variation.
PCA In Four Steps
C = XᵀX / (n−1) — a d×d matrix of how features vary together.C·v = λ·v. Each eigenvector v is a principal direction; its eigenvalue λ is the variance along it. Sort by λ, largest first.Z = X·W → the reduced n×k data.The whole method reduces to one linear-algebra fact: the eigenvectors of the covariance matrix point along the axes of greatest spread, and their eigenvalues measure exactly how much variance each one captures. Sort, keep the top few, project.
The PCA Pipeline, End To End
Standardize so no feature dominates by scale, build the covariance to see how features move together, eigen-decompose to find the principal directions, keep the top k, and project. Out comes a dataset with the same rows but a handful of powerful new columns.
PCA In Four Formulas
Because C is symmetric, its eigenvectors are mutually perpendicular — so the new PC axes are uncorrelated. That's why PCA also wipes out multicollinearity: the components have, by design, zero correlation with each other.
Two Features → One "Ability" Axis
Five students, each with a Maths and a Physics score. The two are highly correlated — strong students tend to do well in both. What does PCA find?
Because Maths and Physics rise and fall together, their real information is essentially one-dimensional — an underlying "ability." PCA discovers that automatically: PC1 explains 99.4% of the variance, so keeping just PC1 loses almost nothing.
Choosing k — The Scree Plot
Rank the components by height like a mountain ridge: the first few are towering peaks, then there's a sharp drop to gentle foothills. The elbow — where the bars flatten and the cumulative line crosses ~95% — tells you how many components to keep.
Three Ways To Pick The Number Of Components
Always inspect the explained_variance_ratio_ before committing. Keeping too few components
throws away signal; keeping too many defeats the purpose. Let the retained-variance number justify your
choice of k.
Standardization Is Non-Negotiable
PCA chases variance — and variance is measured in the feature's own units. If salary
ranges 0–100,000 and age ranges 0–100, salary's raw variance dwarfs age's, so PC1 would
point almost entirely along salary regardless of which feature actually carries the signal. Standardizing
to zero mean and unit variance puts every feature on equal footing first.
PCA With scikit-learn
from sklearn.decomposition import PCA from sklearn.preprocessing import StandardScaler X_scaled = StandardScaler().fit_transform(X) # step 1 — always scale first pca = PCA(n_components=2) # or 0.95 for a variance target X_2d = pca.fit_transform(X_scaled) print(pca.explained_variance_ratio_) # variance per component print(pca.components_) # eigenvectors = loadings # Iris 4→2: PC1 72.96% · PC2 22.85% → 95.81% retained
components_ holds the loadings — how much each original feature contributes to each PC. On
Iris, PC1 loads heavily on petal length (+0.58), petal width (+0.57) and sepal length (+0.52), so PC1
reads as an overall "flower size" axis. Loadings are how you interpret what a component
means.
Put PCA Inside A Pipeline
from sklearn.pipeline import Pipeline from sklearn.svm import SVC pipe = Pipeline([ ('scaler', StandardScaler()), ('pca', PCA(n_components=0.95)), # keep 95% variance ('svm', SVC()) ]) pipe.fit(X_train, y_train) # scaler + PCA refit on each CV fold — no leakage
If you fit PCA on the whole dataset before splitting, information from the test fold leaks into
training and your cross-validation scores become optimistic lies. Wrapping the scaler and PCA in a
Pipeline means they're refit on the training portion of each fold only — leakage-free by
construction. For huge data, swap in IncrementalPCA(batch_size=…) for constant memory.
64 Pixels → 30 Components
The handwritten-digits dataset has 64 pixel features per image. Compress it to just 30 principal components and train a logistic-regression classifier:
Dropping from 64 to 30 features — under half the original — costs only about 0.8% accuracy while nearly halving the model's input size. That's the PCA bargain: big compression, tiny information loss, faster and simpler downstream models.
When PCA Helps — And When It Hurts
| ✅ Reach For PCA When… | ❌ Avoid PCA When… |
|---|---|
| You want to visualize high-D data in 2-D/3-D | You need interpretable, business-meaningful features |
| You need to denoise (reconstruct via Z·Wᵀ) | The structure is non-linear / curved manifolds |
| You must remove multicollinearity (orthogonal PCs) | n ≪ d — covariance is unreliable, noise gets amplified |
| You want faster training on many correlated features | You're using it to "fix" imbalance or missing values |
PCA keeps the directions of greatest spread — which aren't always the directions most useful for your label. For a supervised task where class separation matters more than raw variance, a supervised method like LDA may serve better. Always confirm the downstream metric actually improves.
PCA vs Other Reduction Methods
| Method | Type | Preserves | Best For |
|---|---|---|---|
| PCA | Linear | Global variance | Preprocessing · denoising · viz |
| Kernel PCA | Non-linear | Non-linear variance | Curved manifolds (slow, O(n²)) |
| t-SNE | Non-linear | Local neighbourhoods | Cluster visualization only |
| UMAP | Non-linear | Local + global | Large data · topology |
| Autoencoder | Non-linear | Learned latent | Images · audio · complex data |
| LDA | Linear (supervised) | Class separation | Classification preprocessing |
Start with PCA — it's fast, interpretable through its loadings, and drops cleanly into a leakage-free pipeline. If clusters still aren't visible after PCA, switch to UMAP (or t-SNE) for visualization, and reach for autoencoders when the data is genuinely non-linear.
Seven Rules For Using PCA Well
You Now Understand PCA End-To-End
PCA rotates data onto the axes of greatest variance, ranks them by eigenvalue, and lets you keep only the top few — compressing many features into a compact, uncorrelated handful. Standardize first, keep it in a pipeline, and check the variance you retain.
Try PCA on the Iris and Digits datasets, read the loadings to interpret each component, then compare UMAP and t-SNE for visualization and LDA for supervised reduction. Explore Kernel PCA when your data curves.
🔦 End of tutorial · Press ← to review, or click Restart