Ensemble Learning

Section 01

The Story That Explains Ensemble Learning

📖 Real World Analogy

The Panel of Judges vs the Single Expert

Imagine you need to value a house. You hire one estate agent. She looks at the property, consults her experience, and says £320,000. Maybe she is right. Maybe she has a blind spot about the neighbourhood, or missed the damp in the basement.

Now imagine instead you hire five different experts — an estate agent, a structural engineer, a local market analyst, a mortgage broker, and a neighbour who sold three streets away last month. Their independent estimates: £310k, £330k, £315k, £325k, £318k. Average: £319,600. Much more reliable.

Each expert has different knowledge, different blind spots, different errors. When those errors are uncorrelated, they cancel each other out in the average. No single person needs to be perfect — the group is. This is the entire philosophy of Ensemble Learning.

In machine learning, those "experts" are individual models. The philosophy works because different models make different mistakes. Average them, and the mistakes cancel. The ensemble wins.

💡

The Core Theorem — Why Ensembles Beat Single Models

If you have N models each with error rate ε < 0.5, and their errors are independent, the ensemble's error rate decreases exponentially as N grows. Even weak learners (barely better than random) combined in large numbers converge to near-perfect accuracy. The catch: errors must be uncorrelated. Diverse models are the secret ingredient.

Section 02

The Ensemble Learning Landscape

There are three major families of ensemble methods, each solving the problem differently. Understanding why each family exists requires first understanding the bias–variance tradeoff.

🌳 ENSEMBLE LEARNING — COMPLETE TAXONOMY DIAGRAM

All three families use multiple models but differ fundamentally in how models are trained and combined. Bagging trains in parallel and reduces variance. Boosting trains sequentially and reduces bias. Stacking trains a second-level model to learn the optimal combination.

Section 03

Bias–Variance Tradeoff — Why Ensembles Exist

Every machine learning model's total error can be decomposed into three components. Ensemble methods are direct engineering responses to this decomposition.

Total Error Decomposition

Error = Bias² + Variance + Noise

Bias² = error from wrong assumptions (underfitting).
Variance = error from sensitivity to training data (overfitting).
Noise = irreducible error in the data itself.

Variance of an Ensemble Average

Var(avg) = ρσ² + (1−ρ)/N · σ²

ρ = correlation between models, σ² = per-model variance, N = number of models.
As N → ∞, the second term vanishes. Only ρσ² remains — the irreducible correlated part. Diversity (low ρ) is everything.

Problem	Cause	Ensemble Fix	Method
High Variance	Single deep tree memorises noise	Average many diverse trees → errors cancel	Bagging / Random Forest
High Bias	Single shallow tree underfits	Sequentially correct residual errors	Boosting (AdaBoost / GBM)
Both	No single model architecture is optimal	Learn optimal combination from data	Stacking / Blending

Section 04

Family 1: Bagging — Parallel Diversity

📖 Story

500 Independent Weather Forecasters

A government wants the most accurate rainfall forecast for tomorrow. They give the same historical weather data to 500 independent meteorologists, but each meteorologist only gets a random 80% subset of the records and a random subset of instruments (features) to study.

Each forecaster produces a different model of rainfall — because they saw different data and focused on different signals. Some overfit to heatwaves. Some overfit to coastal patterns. But when all 500 vote on tomorrow's forecast, the individual quirks cancel out and the consensus is remarkably accurate.

This is Bagging (Bootstrap Aggregating). Same algorithm, different random subsets of data and features, parallel training, majority vote.

Bootstrap Sampling

Draw N random samples with replacement from the training set. Each bootstrap sample contains ~63.2% unique rows (the rest are duplicates). The remaining ~36.8% — the Out-of-Bag (OOB) rows — provide a free validation set.

Train Base Learners in Parallel

Train one model on each bootstrap sample independently. All models are identical in architecture but see different data. This is trivially parallelisable — set n_jobs=-1.

Aggregate Predictions

Classification: majority vote. Regression: arithmetic mean. The aggregation is the ensemble — no second model, no learning, just counting or averaging.

Random Forest Adds Feature Randomness

Random Forest is Bagging + random feature subsampling at each split (only √p features are considered per node). This forces tree diversity even when bootstrap samples are similar — the key innovation over plain Bagging.

from sklearn.ensemble import BaggingClassifier
from sklearn.tree import DecisionTreeClassifier
from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split, cross_val_score
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import Pipeline
import numpy as np

# Noisy dataset — single tree would overfit badly
X, y = make_classification(
    n_samples=1000, n_features=20,
    n_informative=10, n_redundant=5,
    flip_y=0.1,          # 10% label noise
    random_state=42
)
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42
)

# Single deep tree — baseline
single_tree = DecisionTreeClassifier(max_depth=None, random_state=42)
single_tree.fit(X_train, y_train)
print(f"Single Tree  — Train: {single_tree.score(X_train, y_train):.3f}  "
      f"Test: {single_tree.score(X_test, y_test):.3f}")

# Bagging: 100 trees, each sees 80% of data (bootstrap)
bag = BaggingClassifier(
    estimator=DecisionTreeClassifier(max_depth=None),
    n_estimators=100,        # number of trees
    max_samples=0.8,         # each tree sees 80% of training rows
    max_features=0.8,        # each tree sees 80% of features
    bootstrap=True,          # sample with replacement
    oob_score=True,          # free validation — out-of-bag rows
    n_jobs=-1,
    random_state=42
)
bag.fit(X_train, y_train)
print(f"Bagging (100) — Train: {bag.score(X_train, y_train):.3f}  "
      f"Test: {bag.score(X_test, y_test):.3f}  "
      f"OOB: {bag.oob_score_:.3f}")

OUTPUT

Single Tree — Train: 1.000 Test: 0.820 Bagging (100) — Train: 1.000 Test: 0.895 OOB: 0.882

Random Forest — The Production Standard

from sklearn.ensemble import RandomForestClassifier
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split, cross_val_score
from sklearn.metrics import classification_report
import numpy as np

cancer = load_breast_cancer()
X, y = cancer.data, cancer.target

X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42, stratify=y
)

rf = RandomForestClassifier(
    n_estimators=300,       # number of trees
    max_features='sqrt',    # sqrt(p) features per split — the key randomness
    max_depth=None,         # grow fully — bias stays low
    min_samples_leaf=1,
    bootstrap=True,
    oob_score=True,
    n_jobs=-1,
    random_state=42
)
rf.fit(X_train, y_train)

print(f"OOB Score  : {rf.oob_score_:.4f}")
print(f"Test Score : {rf.score(X_test, y_test):.4f}")
print(classification_report(y_test, rf.predict(X_test),
                            target_names=cancer.target_names))

# Feature importances (MDI)
importances = rf.feature_importances_
top5 = np.argsort(importances)[::-1][:5]
print("Top-5 features:")
for i in top5:
    print(f"  {cancer.feature_names[i]:<30} {importances[i]:.4f}")

OUTPUT

OOB Score : 0.9780 Test Score : 0.9737 precision recall f1-score support malignant 0.97 0.95 0.96 42 benign 0.97 0.99 0.98 72 accuracy 0.97 114

Section 05

Family 2: Boosting — Sequential Error Correction

📖 Story

The Exam Tutor Who Only Teaches Your Mistakes

Imagine a student preparing for a maths exam. After the first mock test, a tutor reviews only the questions the student got wrong. The student studies those weak areas. Second mock test — better, but new mistakes appear. The tutor again focuses only on the new errors. Repeat for 100 rounds.

Each round, the student gets progressively better at exactly the problems that previously stumped them. The tutor never wastes time on things already mastered — every session targets the current weaknesses.

This is Boosting. Each new model in the sequence focuses on the examples that all previous models got wrong — the hard cases. Collectively, the sequence covers all difficulty levels.

📊 BOOSTING — HOW EACH ROUND CORRECTS THE PREVIOUS

AdaBoost — Adaptive Boosting

from sklearn.ensemble import AdaBoostClassifier
from sklearn.tree import DecisionTreeClassifier
from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split
import numpy as np

X, y = make_classification(
    n_samples=800, n_features=15, n_informative=8,
    random_state=42
)
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42
)

# AdaBoost with stumps (depth-1 trees)
ada = AdaBoostClassifier(
    estimator=DecisionTreeClassifier(max_depth=1),  # weak learner = stump
    n_estimators=200,       # sequential rounds
    learning_rate=0.5,      # shrinks each tree's contribution
    algorithm='SAMME',
    random_state=42
)
ada.fit(X_train, y_train)

print(f"AdaBoost Test Accuracy: {ada.score(X_test, y_test):.4f}")

# Watch accuracy grow with each additional stump
staged = list(ada.staged_score(X_test, y_test))
print(f"After   10 stumps: {staged[9]:.4f}")
print(f"After   50 stumps: {staged[49]:.4f}")
print(f"After  100 stumps: {staged[99]:.4f}")
print(f"After  200 stumps: {staged[199]:.4f}")

OUTPUT

AdaBoost Test Accuracy: 0.9125 After 10 stumps: 0.8313 After 50 stumps: 0.8875 After 100 stumps: 0.9000 After 200 stumps: 0.9125

Gradient Boosting Machine (GBM)

from sklearn.ensemble import GradientBoostingClassifier
from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split, cross_val_score
import numpy as np

X, y = make_classification(
    n_samples=1000, n_features=20, n_informative=12,
    random_state=42
)
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42
)

gb = GradientBoostingClassifier(
    n_estimators=300,        # number of boosting rounds
    learning_rate=0.05,      # shrinkage — smaller = more robust, needs more trees
    max_depth=4,             # tree complexity — keep low (3-6)
    subsample=0.8,           # stochastic gradient boosting (80% rows per tree)
    max_features='sqrt',     # random feature subsampling per split
    min_samples_leaf=5,
    random_state=42
)
gb.fit(X_train, y_train)

cv_scores = cross_val_score(gb, X_train, y_train, cv=5, scoring='accuracy')
print(f"CV Accuracy  : {cv_scores.mean():.4f} +/- {cv_scores.std():.4f}")
print(f"Test Accuracy: {gb.score(X_test, y_test):.4f}")

# Feature importances from GBM
importances = gb.feature_importances_
top3 = np.argsort(importances)[::-1][:3]
print("Top-3 feature indices:", top3, "importances:", importances[top3].round(4))

OUTPUT

CV Accuracy : 0.9213 +/- 0.0182 Test Accuracy: 0.9350

XGBoost — The Competition Standard

import xgboost as xgb
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split, GridSearchCV
from sklearn.metrics import classification_report
import numpy as np

cancer = load_breast_cancer()
X, y = cancer.data, cancer.target

X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42, stratify=y
)

# XGBoost native API — with early stopping
dtrain = xgb.DMatrix(X_train, label=y_train)
dtest  = xgb.DMatrix(X_test,  label=y_test)

params = {
    'objective':        'binary:logistic',
    'eval_metric':      'logloss',
    'max_depth':        4,
    'learning_rate':    0.05,
    'subsample':        0.8,
    'colsample_bytree': 0.8,   # feature subsampling per tree
    'reg_alpha':        0.1,   # L1 regularisation
    'reg_lambda':       1.0,   # L2 regularisation
    'seed':             42
}

model = xgb.train(
    params, dtrain,
    num_boost_round=500,
    evals=[(dtest, 'test')],
    early_stopping_rounds=30,  # stop if no improvement for 30 rounds
    verbose_eval=50
)

y_pred = (model.predict(dtest) > 0.5).astype(int)
print(classification_report(y_test, y_pred,
                            target_names=cancer.target_names))
print(f"Best iteration: {model.best_iteration}")

OUTPUT

[0] test-logloss: 0.54321 [50] test-logloss: 0.12847 [100] test-logloss: 0.08932 ... Best iteration: 312 precision recall f1-score support malignant 0.98 0.95 0.97 42 benign 0.97 0.99 0.98 72 accuracy 0.97 114

⚡

AdaBoost vs GBM vs XGBoost — Which to Use?

AdaBoost: Simple, fast, good for binary classification with clean data. Sensitive to noise and outliers (they get very high weights).
GBM: More flexible (arbitrary loss functions), smoother, better handles noise via subsample. Slower than AdaBoost.
XGBoost / LightGBM: Same math as GBM but engineered for speed (histogram-based splits, column-wise parallelism, sparsity handling). Use these in production and competitions. Default choice for tabular data.

Section 06

Family 3: Stacking — Learning to Combine

📖 Story

The Five Specialists and the Senior Consultant

A hospital has five specialist doctors: a cardiologist, a neurologist, a radiologist, an endocrinologist, and a geneticist. For a complex patient, each specialist gives their diagnosis. But instead of simple majority vote, the hospital brings in a Senior Consultant — someone who has spent years learning which specialists to trust for which types of cases.

For a patient with ambiguous heart symptoms, she learned that the cardiologist + geneticist combination is most reliable. For diabetes cases, the endocrinologist dominates. She does not vote — she has learned the optimal weighting from thousands of past cases.

That senior consultant is the meta-learner in stacking. The five specialists are the base learners. The genius is that the meta-learner's knowledge was learned from data — not hand-coded.

🌍 STACKING — TWO-LEVEL ARCHITECTURE

Each base learner is trained on K−1 folds and predicts on the held-out fold — generating out-of-fold predictions that form the meta-feature matrix. This prevents leakage: the meta-learner never sees training examples that a base learner was trained on.

from sklearn.ensemble import StackingClassifier, RandomForestClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.svm import SVC
from sklearn.neighbors import KNeighborsClassifier
from sklearn.tree import DecisionTreeClassifier
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split, cross_val_score
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import Pipeline

cancer = load_breast_cancer()
X, y = cancer.data, cancer.target
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42, stratify=y
)

# Level-0: diverse base learners
base_learners = [
    ('rf',  RandomForestClassifier(n_estimators=100, random_state=42, n_jobs=-1)),
    ('svm', Pipeline([('sc', StandardScaler()),
                      ('svc', SVC(probability=True, kernel='rbf', C=10))])),
    ('knn', Pipeline([('sc', StandardScaler()),
                      ('knn', KNeighborsClassifier(n_neighbors=11, weights='distance'))])),
    ('dt',  DecisionTreeClassifier(max_depth=5, random_state=42)),
]

# Level-1: meta-learner (learns from base-learner outputs)
meta = LogisticRegression(C=1.0, max_iter=1000, random_state=42)

stack = StackingClassifier(
    estimators=base_learners,
    final_estimator=meta,
    cv=5,                # 5-fold CV generates meta-features (prevents leakage)
    stack_method='predict_proba',
    n_jobs=-1,
    passthrough=False    # meta-learner only sees predictions, not raw features
)

stack.fit(X_train, y_train)

# Compare all models
for name, model in base_learners + [('stack', stack)]:
    if hasattr(model, 'fit'):
        model.fit(X_train, y_train)
    score = model.score(X_test, y_test)
    print(f"  {name:<8} Test accuracy: {score:.4f}")

OUTPUT

rf Test accuracy: 0.9386 svm Test accuracy: 0.9561 knn Test accuracy: 0.9298 dt Test accuracy: 0.9123 stack Test accuracy: 0.9649

Voting Classifier — The Simplest Stacking

from sklearn.ensemble import VotingClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.svm import SVC
from sklearn.ensemble import RandomForestClassifier
from sklearn.neighbors import KNeighborsClassifier
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import Pipeline
from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split

X, y = make_classification(
    n_samples=1000, n_features=15, n_informative=10, random_state=42
)
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42
)

estimators = [
    ('lr',  Pipeline([('sc', StandardScaler()),
                      ('lr', LogisticRegression(max_iter=1000))])),
    ('svm', Pipeline([('sc', StandardScaler()),
                      ('svm', SVC(probability=True, kernel='rbf'))])),
    ('rf',  RandomForestClassifier(n_estimators=100, random_state=42)),
    ('knn', Pipeline([('sc', StandardScaler()),
                      ('knn', KNeighborsClassifier(n_neighbors=9))])),
]

# Hard voting — each model casts one vote for its predicted class
hard_vote = VotingClassifier(estimators=estimators, voting='hard', n_jobs=-1)

# Soft voting — average predicted probabilities, pick highest
soft_vote = VotingClassifier(estimators=estimators, voting='soft', n_jobs=-1)

for name, clf in [('Hard Voting', hard_vote), ('Soft Voting', soft_vote)]:
    clf.fit(X_train, y_train)
    print(f"{name}: {clf.score(X_test, y_test):.4f}")

OUTPUT

Hard Voting: 0.9350 Soft Voting: 0.9400

🎯

Hard vs Soft Voting — When to Use Each

Hard voting: each model casts one vote for its predicted class. Simple, fast, works even when models don't output probabilities.
Soft voting: average the predicted probabilities, pick the class with the highest mean probability. Almost always better than hard voting because it uses more information (confidence levels, not just the winning class). Use voting='soft' unless your models don't support predict_proba.

Section 07

Head-to-Head Comparison — All Methods on Same Data

import numpy as np
from sklearn.tree import DecisionTreeClassifier
from sklearn.ensemble import (
    BaggingClassifier, RandomForestClassifier,
    AdaBoostClassifier, GradientBoostingClassifier
)
from sklearn.model_selection import cross_val_score
from sklearn.datasets import make_classification
import warnings
warnings.filterwarnings('ignore')

X, y = make_classification(
    n_samples=2000, n_features=20, n_informative=12,
    n_redundant=4, flip_y=0.05, random_state=42
)

models = {
    'Decision Tree' :  DecisionTreeClassifier(max_depth=None, random_state=42),
    'Bagging'       :  BaggingClassifier(n_estimators=100, random_state=42, n_jobs=-1),
    'Random Forest' :  RandomForestClassifier(n_estimators=100, random_state=42, n_jobs=-1),
    'AdaBoost'      :  AdaBoostClassifier(n_estimators=100, random_state=42),
    'Gradient Boost':  GradientBoostingClassifier(n_estimators=100, random_state=42),
}

print(f"{'Model':<18} {'CV Mean':>8} {'CV Std':>8}")
print("-" * 38)
for name, model in models.items():
    scores = cross_val_score(model, X, y, cv=5, scoring='accuracy', n_jobs=-1)
    print(f"{name:<18} {scores.mean():>8.4f} {scores.std():>8.4f}")

OUTPUT

Model CV Mean CV Std -------------------------------------- Decision Tree 0.8042 0.0212 Bagging 0.8836 0.0148 Random Forest 0.8940 0.0121 AdaBoost 0.8648 0.0162 Gradient Boost 0.9027 0.0118

📊 ACCURACY COMPARISON — SINGLE TREE VS ENSEMBLE METHODS

Section 08

Key Hyperparameters Across All Methods

Method	Key Parameters	What They Control	Tuning Priority
Bagging	`n_estimators`, `max_samples`, `max_features`	More trees always help (diminishing returns after ~200). max_samples/features control diversity.	Low — defaults work well
Random Forest	`n_estimators`, `max_features`, `min_samples_leaf`	max_features=‘sqrt’ for classification, 1/3 for regression. Tune min_samples_leaf for noisy data.	Medium — tune max_features first
AdaBoost	`n_estimators`, `learning_rate`	learning_rate shrinks each tree’s contribution. Lower rate needs more estimators.	Medium — tune together
GBM / XGBoost	`n_estimators`, `learning_rate`, `max_depth`, `subsample`	Most sensitive: learning_rate × n_estimators tradeoff. Low lr + more trees = better. max_depth 3–6.	High — use RandomizedSearchCV
Stacking	Base learner diversity, meta-learner choice, `cv` folds	Diverse base learners matter most. Meta-learner is usually simple (LogReg). cv=5 prevents leakage.	Medium — choose diverse bases

Section 09

When to Use Each Method

🌲

Random Forest

Fast, robust, parallelisable, built-in OOB validation, no scaling needed. Use as your first baseline for any tabular classification or regression task. Works well out of the box with zero tuning.

default first choice · tabular data

🚀

XGBoost / LightGBM

Highest accuracy on tabular data. Handles missing values, sparse matrices, custom objectives. Requires careful tuning (learning rate, depth, regularisation). The standard for Kaggle competitions and production ML.

maximum accuracy · competition · tabular

🪵

AdaBoost

Simple and fast. Best for clean, balanced datasets. Avoid when data has significant noise or outliers — they receive very high weights and corrupt the model.

clean data · binary classification

🌟

Stacking

Highest possible accuracy when you have diverse, well-tuned base learners. Complex to implement correctly (must avoid leakage). Use when you have exhausted individual model tuning and need the last 1–2%.

competitions · maximum accuracy · complex setup

⛔

Avoid Ensembles When…

Interpretability is legally required (use a single decision tree instead). Dataset is tiny <200 rows (overfitting risk, no diversity possible). Real-time prediction under 1ms latency is required (single linear model wins).

regulated · tiny data · extreme latency

📋

Voting Classifier

Quick win when you already have several well-tuned models and want a fast ensemble without the complexity of stacking. Use voting='soft' always. Gains are modest but reliable.

quick ensemble · existing models · soft voting

Section 10

Common Pitfalls

Correlated base learners

Using RF + ExtraTrees + Bagging in a stacking ensemble — they all make the same mistakes. Diversity is the entire point. Mix fundamentally different algorithms: tree-based + linear + instance-based.

Data leakage in stacking

Training base learners on the full training set, then generating meta-features from that same set — meta-learner sees artificially perfect predictions. Always use K-fold out-of-fold generation. Use sklearn's StackingClassifier which handles this automatically.

Over-boosting (too many rounds)

GBM/XGBoost with a very low learning rate and thousands of trees will overfit without early stopping. Always monitor validation loss. Use early_stopping_rounds=30 in XGBoost or keep a validation set for GBM.

Evaluating with accuracy on imbalanced data

A fraud dataset with 99% negative class — a model predicting always-negative gets 99% accuracy but zero usefulness. Use F1, AUC-ROC, or average_precision for imbalanced ensembles.

Not scaling before stacking with SVM/KNN

Random Forest doesn't need scaling, but SVM and KNN do. When mixing them in a stacking ensemble, each model must handle its own scaling inside a Pipeline — don't scale the whole dataset once before all models.

Treating MDI feature importance as ground truth

Mean Decrease Impurity (the default feature_importances_ in RF) is biased toward high-cardinality numerical features. Use permutation_importance or SHAP values for reliable attribution when making business decisions.

Section 11

Golden Rules

🌳 Ensemble Learning — Non-Negotiable Rules

Start with Random Forest, not a single tree. For any new tabular ML task, Random Forest is your first model. It requires no scaling, handles mixed data types, provides OOB validation, gives feature importances, and almost always beats a single tree with zero tuning. It is the correct default baseline.

Diversity beats individual accuracy in ensembles. Five mediocre but uncorrelated models beat five excellent but identical models every time. When building stacking or voting ensembles, prioritise architectural diversity: tree-based + linear + distance-based + neural. Same architecture, different hyperparameters is not diversity.

Boosting needs early stopping — always. GBM and XGBoost will overfit with enough rounds. Always hold out a validation set and use early_stopping_rounds in XGBoost or monitor validation loss manually in GBM. The optimal number of rounds is a hyperparameter, not a fixed value.

In stacking, use out-of-fold predictions — never train-set predictions. If base learners are evaluated on data they trained on, their predictions are near-perfect on training rows — the meta-learner learns to trust them unconditionally and overfits. Use StackingClassifier(cv=5) which handles OOF generation automatically.

More trees never hurts Bagging/Random Forest — only costs time. Unlike boosting (which can overfit with more rounds), adding more trees to a Bagging ensemble can only decrease or maintain error — never increase it. The only cost is compute. After ~300–500 trees the gain is negligible, but it never goes negative.

Use n_jobs=-1 on every ensemble. Bagging and Random Forest are trivially parallelisable — each tree is independent. On an 8-core machine, training time drops by ~7× for free. Always set this. Not setting it is leaving compute on the table.

Evaluate ensembles on the right metric for your problem. Accuracy misleads on imbalanced data. F1 ignores true negatives. AUC-ROC ignores calibration. Choose the metric that reflects actual business cost — and evaluate it the same way in cross-validation as on the test set.