Linear Discriminant Analysis (LDA)

The Story That Explains LDAवह कहानी जो LDA समझाती है一个解释 LDA 的故事

Linear Discriminant Analysis (LDA) is a supervised method for dimensionality reduction and classification, introduced by Ronald Fisher in 1936. PCA keeps the directions with the most total variance and never looks at labels. LDA uses the labels and maximises the ratio of between-class scatter to within-class scatter, so it finds the projection that makes the classes easiest to tell apart.

Linear Discriminant Analysis (LDA) dimensionality reduction और classification का एक supervised तरीका है, जिसे Ronald Fisher ने 1936 में दिया। PCA सबसे ज़्यादा कुल variance वाली दिशाएँ रखता है और labels कभी नहीं देखता। LDA labels इस्तेमाल करता है और between-class scatter और within-class scatter के अनुपात को सबसे बड़ा करता है, इसलिए यह वह projection खोजता है जिसमें classes को अलग पहचानना सबसे आसान हो।

线性判别分析（LDA）是一种用于降维和分类的有监督（supervised）方法，由 Ronald Fisher 于 1936 年提出。 PCA 保留总方差最大的方向，从不看标签；LDA 使用标签，最大化类间散布（between-class scatter）与 类内散布（within-class scatter）之比，从而找到最容易区分各类别的投影。

The Core Intuition — Two Objectives at Onceमुख्य समझ — एक साथ दो लक्ष्य核心直觉 — 同时实现两个目标

The LDA Words, in PicturesLDA के शब्द, तस्वीरों में用图片解释 LDA 术语

3-D Visualisation — PCA View vs LDA View3-D Visualisation — PCA View vs LDA View三维可视化 — PCA 视图与 LDA 视图

Here are three classes in 3-D. All three share a big "noise" direction (along x₃). Both PCA and LDA are computed live in your browser from the data. PCA keeps the two directions with the most total spread. LDA keeps the two directions that separate the class means. Switch between them and watch the right panel.

यहाँ 3-D में तीन classes हैं। तीनों में एक बड़ी "noise" दिशा (x₃ के साथ) साझा है। PCA और LDA दोनों आपके browser में data से live निकाले जाते हैं। PCA सबसे ज़्यादा कुल फैलाव वाली दो दिशाएँ रखता है। LDA वे दो दिशाएँ रखता है जो class means को अलग करती हैं। दोनों के बीच बदलें और दाईं panel देखें।

这里是三维空间中的三个类别。它们共有一个很大的"噪声"方向（沿 x₃）。PCA 和 LDA 都由你的浏览器根据数据实时计算。 PCA 保留总散布最大的两个方向；LDA 保留能把类均值分开的两个方向。在两者之间切换，观察右侧面板。

Step-by-Step Maths — A Worked ExampleStep-by-Step गणित — एक हल किया हुआ Example逐步推导 — 一个完整例题

Numerical Mini-Example (2 Features, 2 Classes, 4 Points)छोटा Numerical Example (2 Features, 2 Classes, 4 Points)数值小例题（2 个特征、2 个类别、4 个点）

LDA Assumptions — The Fine PrintLDA की मान्यताएँ — बारीक बातेंLDA 的假设 — 细则

Python Implementation — Wine DatasetPython Implementation — Wine DatasetPython 实现 — 葡萄酒数据集

LDA as a Dimensionality Reducer + VisualiserDimensionality Reducer + Visualiser के रूप में LDALDA 作为降维与可视化工具

from sklearn.discriminant_analysis import LinearDiscriminantAnalysis
from sklearn.datasets import load_wine
from sklearn.preprocessing import StandardScaler
import numpy as np
import matplotlib.pyplot as plt

# Load the UCI Wine dataset — 178 samples, 13 chemical features, 3 classes
wine = load_wine()
X, y = wine.data, wine.target
labels = wine.target_names  # ['class_0', 'class_1', 'class_2']

# Standardise features (recommended for LDA)
scaler = StandardScaler()
X_scaled = scaler.fit_transform(X)

# Fit LDA — project down to 2 components (C-1 = 3-1 = 2 max)
lda = LinearDiscriminantAnalysis(n_components=2)
X_lda = lda.fit_transform(X_scaled, y)

# Explained variance ratio of each discriminant
print("Explained variance ratio:", lda.explained_variance_ratio_)
print(f"LD1 explains: {lda.explained_variance_ratio_[0]*100:.1f}%")
print(f"LD2 explains: {lda.explained_variance_ratio_[1]*100:.1f}%")

# Plot the 2D projection
colors = ['#f87171', '#60a5fa', '#34d399']
plt.figure(figsize=(8,6))
for i, label in enumerate(labels):
    mask = y == i
    plt.scatter(X_lda[mask, 0], X_lda[mask, 1],
                label=label, color=colors[i], alpha=0.8, edgecolors='white', s=60)
plt.xlabel('LD1'); plt.ylabel('LD2')
plt.title('Wine Dataset — LDA 2D Projection')
plt.legend(); plt.tight_layout(); plt.show()

LDA as a ClassifierClassifier के रूप में LDALDA 作为分类器

from sklearn.discriminant_analysis import LinearDiscriminantAnalysis
from sklearn.model_selection import cross_val_score, StratifiedKFold
from sklearn.metrics import classification_report, confusion_matrix
from sklearn.datasets import load_wine
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import Pipeline

# Build a proper pipeline: scale → LDA classifier
pipe = Pipeline([
    ('scaler', StandardScaler()),
    ('lda',    LinearDiscriminantAnalysis())   # n_components=None → uses all C-1
])

wine = load_wine()
X, y = wine.data, wine.target

# 5-fold stratified cross-validation
cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)
scores = cross_val_score(pipe, X, y, cv=cv, scoring='accuracy')

print(f"CV Accuracy: {scores.mean():.4f} +/- {scores.std():.4f}")

# Full fit + detailed report
from sklearn.model_selection import train_test_split
X_tr, X_te, y_tr, y_te = train_test_split(
    X, y, test_size=0.25, stratify=y, random_state=42
)
pipe.fit(X_tr, y_tr)
y_pred = pipe.predict(X_te)

print(classification_report(y_te, y_pred, target_names=wine.target_names))
print("Confusion Matrix:")
print(confusion_matrix(y_te, y_pred))

Regularised LDA — Handling Small Samples & Many FeaturesRegularised LDA — कम Samples और ज़्यादा Features正则化 LDA — 应对小样本与高维特征

Standard LDA struggles when there are many features and few samples: S_W is estimated badly (or is singular). Regularised LDA adds a shrinkage penalty that pulls the within-class covariance toward a simple diagonal matrix:

जब features बहुत और samples कम हों, तो साधारण LDA मुश्किल में पड़ता है: S_W ठीक से estimate नहीं होता (या singular होता है)। Regularised LDA एक shrinkage penalty जोड़ता है जो within-class covariance को एक सरल diagonal matrix की ओर खींचती है:

特征多、样本少时，标准 LDA 会遇到困难：S_W 估计得很差（甚至是奇异的）。 正则化 LDA 加入一个收缩惩罚，把类内协方差拉向一个简单的对角矩阵：

from sklearn.discriminant_analysis import LinearDiscriminantAnalysis
from sklearn.datasets import make_classification
from sklearn.model_selection import cross_val_score
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import Pipeline

# Simulate high-dimensional, small-sample scenario
# 200 samples, 150 features — classic small-n-large-p regime
X, y = make_classification(
    n_samples=200, n_features=150, n_informative=30,
    n_classes=3, n_clusters_per_class=1, random_state=42
)

# Standard LDA — likely to fail or be unstable
std_pipe = Pipeline([
    ('sc', StandardScaler()),
    ('lda', LinearDiscriminantAnalysis(solver='svd'))    # SVD-based, no shrinkage
])

# Regularised LDA — Ledoit-Wolf automatic shrinkage
reg_pipe = Pipeline([
    ('sc', StandardScaler()),
    ('lda', LinearDiscriminantAnalysis(
        solver='lsqr',
        shrinkage='auto'   # Ledoit-Wolf optimal shrinkage
    ))
])

std_scores = cross_val_score(std_pipe, X, y, cv=5, scoring='accuracy')
reg_scores = cross_val_score(reg_pipe, X, y, cv=5, scoring='accuracy')

print(f"Standard LDA CV:    {std_scores.mean():.4f} +/- {std_scores.std():.4f}")
print(f"Regularised LDA CV: {reg_scores.mean():.4f} +/- {reg_scores.std():.4f}")

# Manual shrinkage tuning (grid search over λ)
from sklearn.model_selection import GridSearchCV
param_grid = {'lda__shrinkage': [0.0, 0.1, 0.3, 0.5, 0.7, 0.9, 1.0, 'auto']}
gs = GridSearchCV(reg_pipe, param_grid, cv=5, scoring='accuracy')
gs.fit(X, y)
print(f"Best shrinkage: {gs.best_params_}")
print(f"Best CV accuracy: {gs.best_score_:.4f}")

Real-World Example — Face Recognition (Fisherfaces)Real-World Example — चेहरा पहचानना (Fisherfaces)实战案例 — 人脸识别（Fisherfaces）

from sklearn.datasets import fetch_olivetti_faces
from sklearn.discriminant_analysis import LinearDiscriminantAnalysis
from sklearn.decomposition import PCA
from sklearn.pipeline import Pipeline
from sklearn.model_selection import cross_val_score, StratifiedKFold
from sklearn.preprocessing import StandardScaler

# Olivetti Faces: 400 images, 4096 features, 40 classes
data = fetch_olivetti_faces(shuffle=True, random_state=42)
X, y = data.data, data.target  # X shape: (400, 4096)

# Fisherfaces pipeline: PCA (Eigenfaces) → LDA (Fisherfaces)
fisherfaces = Pipeline([
    ('scale', StandardScaler()),
    ('pca',   PCA(n_components=150, whiten=True, random_state=42)),
    ('lda',   LinearDiscriminantAnalysis())   # max 39 components (40 classes - 1)
])

cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)
scores = cross_val_score(fisherfaces, X, y, cv=cv, scoring='accuracy')

print("=== Fisherfaces (PCA → LDA) ===")
print(f"CV Accuracy: {scores.mean()*100:.1f}% +/- {scores.std()*100:.1f}%")
print(f"Input dims:  4096 → PCA: 150 → LDA: 39")

# Compare: PCA-only nearest centroid
from sklearn.neighbors import NearestCentroid
pca_only = Pipeline([
    ('scale', StandardScaler()),
    ('pca',   PCA(n_components=150, whiten=True, random_state=42)),
    ('clf',   NearestCentroid())
])
pca_scores = cross_val_score(pca_only, X, y, cv=cv, scoring='accuracy')
print("\n=== PCA-only nearest centroid ===")
print(f"CV Accuracy: {pca_scores.mean()*100:.1f}% +/- {pca_scores.std()*100:.1f}%")

from sklearn.datasets import load_digits
from sklearn.discriminant_analysis import LinearDiscriminantAnalysis
from sklearn.decomposition import PCA
from sklearn.neighbors import NearestCentroid
from sklearn.pipeline import Pipeline
from sklearn.model_selection import cross_val_score, StratifiedKFold
from sklearn.preprocessing import StandardScaler

# Same idea, runs offline: 1,797 digit images (8×8 = 64 pixels), 10 classes
X, y = load_digits(return_X_y=True)
cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)

pca_only = Pipeline([
    ('scale', StandardScaler()),
    ('pca',   PCA(n_components=30, random_state=42)),
    ('clf',   NearestCentroid()),            # PCA alone, then nearest class mean
])
pca_lda = Pipeline([
    ('scale', StandardScaler()),
    ('pca',   PCA(n_components=30, random_state=42)),
    ('lda',   LinearDiscriminantAnalysis()), # max 9 components (10 classes - 1)
])

for name, model in [("PCA-only nearest centroid", pca_only), ("PCA -> LDA (Fisher)", pca_lda)]:
    s = cross_val_score(model, X, y, cv=cv, scoring='accuracy')
    print(f"{name:26s}: {s.mean()*100:.1f}% +/- {s.std()*100:.1f}%")
print("Dims: 64 -> PCA: 30 -> LDA: 9")

LDA vs PCA — When to Use EachLDA vs PCA — कब कौन साLDA 与 PCA — 各自何时使用

LDA vs QDA — Covariance AssumptionsLDA vs QDA — Covariance की मान्यताएँLDA 与 QDA — 协方差假设

When LDA's equal-covariance assumption fails, Quadratic Discriminant Analysis (QDA) can help. QDA estimates a separate covariance matrix Σ_c for each class, which gives quadratic (curved) decision boundaries instead of straight ones.

जब LDA की बराबर-covariance वाली मान्यता टूटती है, तो Quadratic Discriminant Analysis (QDA) मदद कर सकता है। QDA हर class के लिए अलग covariance matrix Σ_c निकालता है, जिससे सीधी की जगह quadratic (मुड़ी हुई) decision सीमाएँ बनती हैं।

当 LDA 的等协方差假设不成立时，二次判别分析（QDA）可以帮忙。QDA 为每个类单独估计协方差矩阵 Σ_c， 得到的是二次（弯曲）的决策边界，而不是直线。

from sklearn.discriminant_analysis import LinearDiscriminantAnalysis, QuadraticDiscriminantAnalysis
from sklearn.datasets import make_classification
from sklearn.model_selection import cross_val_score
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import Pipeline

# Generate data with UNEQUAL class covariances — LDA assumption violated
import numpy as np
np.random.seed(42)

X_c1 = np.random.multivariate_normal([0,0], [[1,0],[0,1]], 200)     # circular
X_c2 = np.random.multivariate_normal([3,3], [[4,2],[2,4]], 200)     # elongated ellipse
X_c3 = np.random.multivariate_normal([6,0], [[0.5,0],[0,3]],200)    # narrow tall

X = np.vstack([X_c1, X_c2, X_c3])
y = np.array([0]*200 + [1]*200 + [2]*200)

lda_pipe = Pipeline([('sc', StandardScaler()), ('clf', LinearDiscriminantAnalysis())])
qda_pipe = Pipeline([('sc', StandardScaler()), ('clf', QuadraticDiscriminantAnalysis())])

lda_cv = cross_val_score(lda_pipe, X, y, cv=5).mean()
qda_cv = cross_val_score(qda_pipe, X, y, cv=5).mean()

print(f"LDA CV Accuracy: {lda_cv:.4f}  (assumption violated)")
print(f"QDA CV Accuracy: {qda_cv:.4f}  (no equal-covariance assumption)")

Medical Diagnosis Example — Breast Cancer LDAMedical Diagnosis Example — Breast Cancer LDA医学诊断案例 — 乳腺癌 LDA

from sklearn.datasets import load_breast_cancer
from sklearn.discriminant_analysis import LinearDiscriminantAnalysis
from sklearn.model_selection import cross_val_score, StratifiedKFold
from sklearn.metrics import roc_auc_score, classification_report
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.model_selection import train_test_split
import numpy as np

bc = load_breast_cancer()
X, y = bc.data, bc.target   # 569 samples, 30 features, 2 classes

pipe = Pipeline([
    ('sc',  StandardScaler()),
    ('lda', LinearDiscriminantAnalysis())
])

# 10-fold stratified CV — important for medical data
cv = StratifiedKFold(n_splits=10, shuffle=True, random_state=42)
acc_scores = cross_val_score(pipe, X, y, cv=cv, scoring='accuracy')
auc_scores = cross_val_score(pipe, X, y, cv=cv, scoring='roc_auc')

print(f"CV Accuracy: {acc_scores.mean():.4f} +/- {acc_scores.std():.4f}")
print(f"CV AUC-ROC:  {auc_scores.mean():.4f} +/- {auc_scores.std():.4f}")

# Full fit for feature weights (discriminant coefficients)
pipe.fit(X, y)
coef = pipe.named_steps['lda'].coef_[0]
top_idx = np.argsort(np.abs(coef))[::-1][:5]
print("\nTop 5 coefficients - plain LDA (signed):")
for i in top_idx:
    print(f"  {bc.feature_names[i]:25s}: {coef[i]:+8.2f}")

# Same model with shrinkage: much smaller, more stable weights
shr = Pipeline([('sc', StandardScaler()),
                ('lda', LinearDiscriminantAnalysis(solver='lsqr', shrinkage='auto'))]).fit(X, y)
coef_s = shr.named_steps['lda'].coef_[0]
print("\nTop 5 coefficients - shrinkage LDA (signed):")
for i in np.argsort(np.abs(coef_s))[::-1][:5]:
    print(f"  {bc.feature_names[i]:25s}: {coef_s[i]:+8.2f}")
print(f"\nShrinkage LDA CV accuracy: {cross_val_score(shr, X, y, cv=cv).mean():.4f}")

LDA Solvers in Scikit-LearnScikit-Learn में LDA SolversScikit-Learn 中的 LDA 求解器

LDA for Multi-Class — IrisMulti-Class के लिए LDA — Iris多类别 LDA — 鸢尾花

from sklearn.discriminant_analysis import LinearDiscriminantAnalysis
from sklearn.datasets import load_iris
from sklearn.preprocessing import StandardScaler
from sklearn.model_selection import cross_val_score
from sklearn.pipeline import Pipeline
import numpy as np

iris = load_iris()
X, y = iris.data, iris.target   # 150 samples, 4 features, 3 classes

pipe = Pipeline([
    ('sc',  StandardScaler()),
    ('lda', LinearDiscriminantAnalysis(n_components=2))
])

# Cross-validated accuracy
scores = cross_val_score(pipe, X, y, cv=10)
print(f"CV Accuracy: {scores.mean():.4f} +/- {scores.std():.4f}")

# Fit and show explained variance per discriminant
pipe.fit(X, y)
lda = pipe.named_steps['lda']
print(f"\nLD1 explains {lda.explained_variance_ratio_[0]*100:.1f}% of between-class variance")
print(f"LD2 explains {lda.explained_variance_ratio_[1]*100:.1f}% of between-class variance")

# Class separation quality: Mahalanobis-like distances between centroids in LDA space
X_lda = lda.transform(pipe.named_steps['sc'].transform(X))
centroids = [X_lda[y==i].mean(axis=0) for i in range(3)]
for i in range(3):
  for j in range(i+1, 3):
    d = np.linalg.norm(np.array(centroids[i])-np.array(centroids[j]))
    print(f"Distance {iris.target_names[i]} ↔ {iris.target_names[j]}: {d:.3f}")

When to Use (and Avoid) LDALDA कब इस्तेमाल करें (और कब नहीं)什么时候用（以及不用）LDA

Golden Rulesसुनहरे नियम黄金法则