Ensemble Learning — Bagging, Boosting and Stacking

The Story That Explains Ensemble Learningवह कहानी जो Ensemble Learning समझाती है一个解释 Ensemble Learning 的故事

The Ensemble Learning LandscapeEnsemble Learning का नक्शाEnsemble Learning 全景

There are three big families of ensemble methods. Each one combines models in a different way. To see why each family exists, we first need the bias–variance tradeoff (Section 03).

Ensemble methods के तीन बड़े परिवार हैं। हर परिवार models को अलग तरीके से जोड़ता है। हर परिवार क्यों है, यह समझने के लिए पहले bias–variance tradeoff समझना होगा (Section 03)।

Ensemble 方法有三大家族，每个家族用不同的方式组合 model。要理解每个家族为什么存在，先要理解 bias–variance tradeoff（第 03 节）。

Key Differences at a Glanceमुख्य अंतर एक नज़र में主要区别一览

Bias–Variance Tradeoff — Why Ensembles ExistBias–Variance Tradeoff — Ensembles क्यों हैंBias–Variance Tradeoff — Ensemble 为什么存在

A model's total error has three parts. Each ensemble family attacks one of them.

किसी model की कुल error के तीन हिस्से होते हैं। हर ensemble परिवार इनमें से एक पर हमला करता है।

model 的总误差由三部分组成。每个 ensemble 家族针对其中一部分。

Family 1: Bagging — Parallel DiversityFamily 1: Bagging — Parallel Diversity家族 1：Bagging — 并行多样性

from sklearn.ensemble import BaggingClassifier
from sklearn.tree import DecisionTreeClassifier
from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split

# Noisy dataset: a single deep tree will overfit badly
X, y = make_classification(
    n_samples=1000, n_features=20,
    n_informative=10, n_redundant=5,
    flip_y=0.1,              # 10% label noise
    random_state=42
)
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42
)

# Baseline: one fully grown tree
single_tree = DecisionTreeClassifier(max_depth=None, random_state=42)
single_tree.fit(X_train, y_train)
print(f"Single Tree   — Train: {single_tree.score(X_train, y_train):.3f}  "
      f"Test: {single_tree.score(X_test, y_test):.3f}")

# Bagging: 100 trees, each on its own bootstrap sample
bag = BaggingClassifier(
    estimator=DecisionTreeClassifier(max_depth=None),
    n_estimators=100,        # number of trees
    max_samples=0.8,         # each bootstrap sample = 80% of the rows
    max_features=0.8,        # each tree sees 80% of the features
    bootstrap=True,          # sample rows WITH replacement
    oob_score=True,          # free validation on out-of-bag rows
    n_jobs=-1,
    random_state=42
)
bag.fit(X_train, y_train)
print(f"Bagging (100) — Train: {bag.score(X_train, y_train):.3f}  "
      f"Test: {bag.score(X_test, y_test):.3f}  OOB: {bag.oob_score_:.3f}")

The single tree scores 100% on training data but only 73.5% on test data. It memorised the noise. Bagging keeps the same deep trees but averages 100 of them: test accuracy jumps to 88.0%. The OOB score (84.4%) is a fair estimate computed without any test data.

एक अकेला tree training data पर 100% पर test data पर सिर्फ़ 73.5% लाता है। उसने noise रट लिया। Bagging वही deep trees रखता है पर उनमें से 100 का average लेता है: test accuracy 88.0% तक पहुँच जाती है। OOB score (84.4%) बिना किसी test data के निकाला गया एक उचित अनुमान है।

单棵树在训练集上 100%，在测试集上只有 73.5%，它记住了噪声。Bagging 使用同样的深树，但平均了 100 棵：测试 accuracy 跳到 88.0%。OOB 分数（84.4%）是不用任何测试数据得到的合理估计。

Random Forest — The Production StandardRandom Forest — Production का StandardRandom Forest — 生产环境标准

from sklearn.ensemble import RandomForestClassifier
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split
from sklearn.metrics import classification_report
import numpy as np

cancer = load_breast_cancer()
X, y = cancer.data, cancer.target
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42, stratify=y
)

rf = RandomForestClassifier(
    n_estimators=300,        # number of trees
    max_features='sqrt',     # sqrt(p) features per split: the key randomness
    max_depth=None,          # grow fully, so bias stays low
    min_samples_leaf=1,
    bootstrap=True,
    oob_score=True,
    n_jobs=-1,
    random_state=42
)
rf.fit(X_train, y_train)

print(f"OOB Score  : {rf.oob_score_:.4f}")
print(f"Test Score : {rf.score(X_test, y_test):.4f}")
print(classification_report(y_test, rf.predict(X_test),
                            target_names=cancer.target_names))

# Feature importances (MDI)
importances = rf.feature_importances_
top5 = np.argsort(importances)[::-1][:5]
print("Top-5 features:")
for i in top5:
    print(f"  {cancer.feature_names[i]:<24} {importances[i]:.4f}")

Family 2: Boosting — Sequential Error CorrectionFamily 2: Boosting — एक-एक करके गलतियाँ सुधारना家族 2：Boosting — 逐步纠错

AdaBoost — Adaptive BoostingAdaBoost — Adaptive BoostingAdaBoost — 自适应提升

from sklearn.ensemble import AdaBoostClassifier
from sklearn.tree import DecisionTreeClassifier
from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split

X, y = make_classification(
    n_samples=800, n_features=15, n_informative=8, random_state=42
)
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42
)

# AdaBoost with stumps (depth-1 trees)
ada = AdaBoostClassifier(
    estimator=DecisionTreeClassifier(max_depth=1),   # weak learner = stump
    n_estimators=200,        # sequential rounds
    learning_rate=0.5,       # shrinks each stump's vote
    random_state=42
)
ada.fit(X_train, y_train)
print(f"AdaBoost Test Accuracy: {ada.score(X_test, y_test):.4f}")

# Watch accuracy grow as stumps are added
staged = list(ada.staged_score(X_test, y_test))
for n in [1, 10, 50, 100, 200]:
    print(f"Rounds {n:>3}: {staged[n - 1]:.4f}")

Accuracy climbs from 60.6% with one stump to 83.1% with 200. Each stump is weak on its own. Together they are strong.

एक stump के साथ accuracy 60.6% से बढ़कर 200 stumps पर 83.1% हो जाती है। हर stump अकेले कमज़ोर है। साथ मिलकर वे मज़बूत हैं।

accuracy 从 1 个 stump 时的 60.6% 上升到 200 个时的 83.1%。每个 stump 单独都很弱，合起来就很强。

Gradient Boosting Machine (GBM)Gradient Boosting Machine (GBM)Gradient Boosting Machine (GBM)

from sklearn.ensemble import GradientBoostingClassifier
from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split, cross_val_score
import numpy as np

X, y = make_classification(
    n_samples=1000, n_features=20, n_informative=12, random_state=42
)
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42
)

gb = GradientBoostingClassifier(
    n_estimators=300,        # number of boosting rounds
    learning_rate=0.05,      # shrinkage: smaller = safer, needs more trees
    max_depth=4,             # keep trees small (3-6)
    subsample=0.8,           # stochastic boosting: 80% rows per tree
    max_features='sqrt',     # random features per split
    min_samples_leaf=5,
    random_state=42
)
gb.fit(X_train, y_train)

cv_scores = cross_val_score(gb, X_train, y_train, cv=5, scoring='accuracy')
print(f"CV Accuracy  : {cv_scores.mean():.4f} +/- {cv_scores.std():.4f}")
print(f"Test Accuracy: {gb.score(X_test, y_test):.4f}")

top3 = np.argsort(gb.feature_importances_)[::-1][:3]
print("Top-3 feature indices:", top3,
      "importances:", gb.feature_importances_[top3].round(4))

XGBoost — The Competition StandardXGBoost — Competitions का StandardXGBoost — 竞赛标准

import xgboost as xgb
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split
from sklearn.metrics import classification_report

cancer = load_breast_cancer()
X, y = cancer.data, cancer.target
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42, stratify=y
)
# Early stopping needs its OWN validation set.
# Never stop on the test set: that leaks test data into training.
X_tr, X_val, y_tr, y_val = train_test_split(
    X_train, y_train, test_size=0.2, random_state=42, stratify=y_train
)

dtrain = xgb.DMatrix(X_tr,  label=y_tr)
dval   = xgb.DMatrix(X_val, label=y_val)
dtest  = xgb.DMatrix(X_test)

params = {
    'objective':        'binary:logistic',
    'eval_metric':      'logloss',
    'max_depth':        4,
    'learning_rate':    0.05,
    'subsample':        0.8,
    'colsample_bytree': 0.8,   # feature subsampling per tree
    'reg_alpha':        0.1,   # L1 regularisation
    'reg_lambda':       1.0,   # L2 regularisation
    'seed':             42
}

model = xgb.train(
    params, dtrain,
    num_boost_round=500,
    evals=[(dval, 'val')],
    early_stopping_rounds=30,  # stop if no improvement for 30 rounds
    verbose_eval=50
)
print(f"Best iteration: {model.best_iteration}")

proba = model.predict(dtest, iteration_range=(0, model.best_iteration + 1))
y_pred = (proba > 0.5).astype(int)
print(classification_report(y_test, y_pred, target_names=cancer.target_names))

Family 3: Stacking — Learning to CombineFamily 3: Stacking — जोड़ना सीखना家族 3：Stacking — 学习如何组合

from sklearn.ensemble import StackingClassifier, RandomForestClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.svm import SVC
from sklearn.neighbors import KNeighborsClassifier
from sklearn.tree import DecisionTreeClassifier
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import Pipeline

cancer = load_breast_cancer()
X, y = cancer.data, cancer.target
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42, stratify=y
)

# Level 0: diverse base learners (scaling lives inside each Pipeline)
base_learners = [
    ('rf',  RandomForestClassifier(n_estimators=100, random_state=42, n_jobs=-1)),
    ('svm', Pipeline([('sc', StandardScaler()),
                      ('svc', SVC(probability=True, C=10, random_state=42))])),
    ('knn', Pipeline([('sc', StandardScaler()),
                      ('knn', KNeighborsClassifier(n_neighbors=11, weights='distance'))])),
    ('dt',  DecisionTreeClassifier(max_depth=5, random_state=42)),
]

# Level 1: meta-learner learns from the base learners' predictions
stack = StackingClassifier(
    estimators=base_learners,
    final_estimator=LogisticRegression(max_iter=1000),
    cv=5,                    # 5-fold out-of-fold meta-features (no leakage)
    stack_method='predict_proba',
    passthrough=False,       # meta-learner sees only predictions
    n_jobs=-1
)

# Compare each base learner with the stack
for name, model in base_learners + [('stack', stack)]:
    model.fit(X_train, y_train)
    print(f"  {name:<6} Test accuracy: {model.score(X_test, y_test):.4f}")

The stack (98.25%) beats every single base learner, including the best ones (SVM and KNN at 97.37%). It works because the four models are very different: trees, a kernel method and a distance method.

Stack (98.25%) हर एक base learner को हराता है, सबसे अच्छे वालों (SVM और KNN, 97.37%) को भी। यह इसलिए काम करता है क्योंकि चारों models बहुत अलग हैं: trees, एक kernel method और एक distance method.

stack（98.25%）胜过了每一个 base learner，包括最好的 SVM 和 KNN（97.37%）。之所以有效，是因为这四个 model 差别很大：树模型、核方法和基于距离的方法。

Voting Classifier — The Simplest CombinationVoting Classifier — सबसे सरल तरीकाVoting Classifier — 最简单的组合方式

from sklearn.ensemble import VotingClassifier, RandomForestClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.svm import SVC
from sklearn.neighbors import KNeighborsClassifier
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import Pipeline
from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split

X, y = make_classification(
    n_samples=1000, n_features=15, n_informative=10, random_state=42
)
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42
)

estimators = [
    ('lr',  Pipeline([('sc', StandardScaler()),
                      ('lr', LogisticRegression(max_iter=1000))])),
    ('svm', Pipeline([('sc', StandardScaler()),
                      ('svm', SVC(probability=True, random_state=42))])),
    ('rf',  RandomForestClassifier(n_estimators=100, random_state=42)),
    ('knn', Pipeline([('sc', StandardScaler()),
                      ('knn', KNeighborsClassifier(n_neighbors=9))])),
]

# Each model on its own
for name, clf in estimators:
    clf.fit(X_train, y_train)
    print(f"{name:<12}: {clf.score(X_test, y_test):.4f}")

hard_vote = VotingClassifier(estimators=estimators, voting='hard', n_jobs=-1)
soft_vote = VotingClassifier(estimators=estimators, voting='soft', n_jobs=-1)
for name, clf in [('Hard Voting', hard_vote), ('Soft Voting', soft_vote)]:
    clf.fit(X_train, y_train)
    print(f"{name:<12}: {clf.score(X_test, y_test):.4f}")

Head-to-Head Comparison — All Methods on the Same Dataआमने-सामने — एक ही Data पर सभी Methods正面对比 — 同一数据上的所有方法

from sklearn.tree import DecisionTreeClassifier
from sklearn.ensemble import (
    BaggingClassifier, RandomForestClassifier,
    AdaBoostClassifier, GradientBoostingClassifier
)
from sklearn.model_selection import cross_val_score
from sklearn.datasets import make_classification

X, y = make_classification(
    n_samples=2000, n_features=20, n_informative=12,
    n_redundant=4, flip_y=0.05, random_state=42
)

models = {
    'Decision Tree' : DecisionTreeClassifier(random_state=42),
    'Bagging'       : BaggingClassifier(n_estimators=100, random_state=42, n_jobs=-1),
    'Random Forest' : RandomForestClassifier(n_estimators=100, random_state=42, n_jobs=-1),
    'AdaBoost'      : AdaBoostClassifier(n_estimators=100, random_state=42),
    'Gradient Boost': GradientBoostingClassifier(n_estimators=100, random_state=42),
}

print(f"{'Model':<16} {'CV Mean':>8} {'CV Std':>8}")
print("-" * 34)
for name, model in models.items():
    scores = cross_val_score(model, X, y, cv=5, scoring='accuracy', n_jobs=-1)
    print(f"{name:<16} {scores.mean():>8.4f} {scores.std():>8.4f}")

On this data, Random Forest wins (91.1%), and every ensemble beats the single tree. Gradient Boosting here uses default settings (100 trees, no tuning). With tuning it often takes the lead. The lesson: the ranking depends on the data and on tuning. Always test a few methods with cross-validation.

इस data पर Random Forest जीतता है (91.1%), और हर ensemble एक tree को हराता है। यहाँ Gradient Boosting default settings पर है (100 trees, कोई tuning नहीं)। Tuning के साथ यह अक्सर आगे निकल जाता है। सबक: ranking data और tuning पर निर्भर करती है। हमेशा cross-validation से कुछ methods test करें।

在这份数据上，Random Forest 胜出（91.1%），而且每个 ensemble 都胜过单棵树。这里的 Gradient Boosting 使用默认设置（100 棵树，未调参），调参后它常常会领先。结论：排名取决于数据和调参。一定要用交叉验证测试几种方法。

Key Hyperparameters Across All Methodsसभी Methods के मुख्य Hyperparameters各方法的关键超参数

When to Use Each Methodकौन-सा Method कब इस्तेमाल करें何时使用哪种方法

Common Pitfallsआम गलतियाँ常见陷阱

Golden RulesGolden Rules黄金法则