Boosting & XGBoost

The Story That Explains Boosting

Boosting is a family of ensemble learning algorithms. Its unifying idea: combine many weak learners (models that are only slightly better than random guessing) into one strong learner by letting each weak learner fix what its predecessors could not. The three most important members of this family are:

AdaBoost — Where It All Started

AdaBoost (Adaptive Boosting) was introduced by Freund & Schapire in 1996 and won the Gödel Prize. It is the simplest boosting algorithm to understand — and understanding it unlocks everything that follows.

The AdaBoost Algorithm — Step by Step

Gradient Boosting — The Generalisation

Gradient Boosting (Friedman, 1999) took AdaBoost's core idea and framed it as gradient descent in function space. Instead of reweighting samples, each new tree directly fits the residual errors (technically, the negative gradients of the loss function) of the current ensemble.

Gradient Boosting — The Algorithm

XGBoost — Extreme Gradient Boosting

What Makes XGBoost Different

The XGBoost Objective Function

Python Implementation — AdaBoost

Let's start with AdaBoost on a classic binary classification problem — predicting whether a bank customer will default on a loan — to see boosting in its simplest form.

import numpy as np
import pandas as pd
from sklearn.ensemble import AdaBoostClassifier
from sklearn.tree import DecisionTreeClassifier
from sklearn.model_selection import train_test_split, cross_val_score
from sklearn.metrics import classification_report, roc_auc_score
from sklearn.datasets import make_classification

# ── Generate synthetic loan default dataset ───────────────
X, y = make_classification(
    n_samples=5000,
    n_features=15,
    n_informative=10,
    n_redundant=3,
    random_state=42,
    class_sep=0.8
)

X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42, stratify=y
)

# ── Build AdaBoost with decision stumps (depth=1) ─────────
# SAMME is the multi-class generalisation of the original AdaBoost
base_estimator = DecisionTreeClassifier(max_depth=1)

ada = AdaBoostClassifier(
    estimator=base_estimator,
    n_estimators=200,       # number of stumps / boosting rounds
    learning_rate=0.5,      # shrinks each stump's contribution
    algorithm='SAMME',      # discrete boosting (original Freund & Schapire)
    random_state=42
)

# ── Cross-validation ──────────────────────────────────────
cv_auc = cross_val_score(ada, X_train, y_train, cv=5, scoring='roc_auc')
print(f"CV ROC-AUC: {cv_auc.mean():.4f} ± {cv_auc.std():.4f}")

# ── Fit and evaluate ──────────────────────────────────────
ada.fit(X_train, y_train)
y_pred  = ada.predict(X_test)
y_proba = ada.predict_proba(X_test)[:, 1]

print(f"\nTest ROC-AUC : {roc_auc_score(y_test, y_proba):.4f}")
print(classification_report(y_test, y_pred))

# ── Inspect individual stump weights (alpha values) ───────
print("\nTop 5 stump weights (α):")
top_idx = np.argsort(ada.estimator_weights_)[::-1][:5]
for i in top_idx:
    print(f"  Stump {i:3d}: α = {ada.estimator_weights_[i]:.4f}")

Python Implementation — XGBoost End-to-End

Now we use the real workhorse. Below is a production-grade XGBoost pipeline on the classic Titanic survival problem, with feature engineering, early stopping, and a full hyperparameter description.

import xgboost as xgb
import pandas as pd
import numpy as np
from sklearn.model_selection import StratifiedKFold, cross_val_score
from sklearn.preprocessing import LabelEncoder
from sklearn.metrics import roc_auc_score

# ── Load data ─────────────────────────────────────────────
df = pd.read_csv('titanic.csv')

# ── Feature engineering ───────────────────────────────────
df['Title'] = df['Name'].str.extract(r' ([A-Za-z]+)\.', expand=False)
df['Title'] = df['Title'].replace(
    {'Mlle':'Miss', 'Ms':'Miss', 'Mme':'Mrs',
     'Lady':'Rare', 'Countess':'Rare', 'Capt':'Rare',
     'Col':'Rare', 'Don':'Rare', 'Dr':'Rare',
     'Major':'Rare', 'Rev':'Rare', 'Sir':'Rare',
     'Jonkheer':'Rare', 'Dona':'Rare'}
)
df['FamilySize'] = df['SibSp'] + df['Parch'] + 1
df['IsAlone']    = (df['FamilySize'] == 1).astype(int)
df['Age'].fillna(df['Age'].median(), inplace=True)
df['Embarked'].fillna(df['Embarked'].mode()[0], inplace=True)
df['Fare'].fillna(df['Fare'].median(), inplace=True)

# Label encode categoricals — XGBoost handles numeric only
for col in ['Sex', 'Embarked', 'Title']:
    df[col] = LabelEncoder().fit_transform(df[col].astype(str))

features = ['Pclass', 'Sex', 'Age', 'SibSp', 'Parch',
            'Fare', 'Embarked', 'Title', 'FamilySize', 'IsAlone']
X = df[features]
y = df['Survived']

# ── XGBoost model — annotated hyperparameters ─────────────
model = xgb.XGBClassifier(

    # ── BOOSTING STRUCTURE ──────────────────────────────
    n_estimators    = 500,      # max trees; use early stopping
    learning_rate   = 0.05,     # η: shrinks each tree's contribution
    max_depth       = 4,        # tree depth: 3-6 is typical

    # ── REGULARISATION ──────────────────────────────────
    reg_alpha       = 0.1,      # L1 on leaf weights (sparsity)
    reg_lambda      = 1.0,      # L2 on leaf weights (smoothing)
    gamma           = 0.05,     # min gain to make a split
    min_child_weight= 3,        # min Hessian sum in a leaf

    # ── RANDOMISATION (like Random Forest) ─────────────
    subsample       = 0.8,      # fraction of rows per tree
    colsample_bytree= 0.7,      # fraction of cols per tree
    colsample_bylevel=0.7,      # fraction of cols per depth level

    # ── PERFORMANCE ─────────────────────────────────────
    tree_method     = 'hist',   # fast histogram-based splits
    device          = 'cpu',    # change to 'cuda' for GPU
    n_jobs          = -1,        # use all CPU cores
    random_state    = 42,
    eval_metric     = 'auc',    # metric for early stopping
    early_stopping_rounds = 30  # stop if no improvement in 30 rounds
)

# ── Fit with a validation set for early stopping ──────────
from sklearn.model_selection import train_test_split
X_tr, X_val, y_tr, y_val = train_test_split(
    X, y, test_size=0.2, random_state=42, stratify=y
)

model.fit(
    X_tr, y_tr,
    eval_set=[(X_val, y_val)],
    verbose=50        # print eval every 50 rounds
)

print(f"\nBest iteration : {model.best_iteration}")
print(f"Best val AUC   : {model.best_score:.4f}")

# ── Cross-validated AUC ───────────────────────────────────
cv_auc = cross_val_score(
    xgb.XGBClassifier(n_estimators=model.best_iteration,
                       learning_rate=0.05, max_depth=4,
                       subsample=0.8, colsample_bytree=0.7,
                       tree_method='hist', n_jobs=-1),
    X, y, cv=StratifiedKFold(5, shuffle=True, random_state=42),
    scoring='roc_auc'
)
print(f"\nCV ROC-AUC: {cv_auc.mean():.4f} ± {cv_auc.std():.4f}")

# ── Feature importance ────────────────────────────────────
importances = pd.Series(model.feature_importances_, index=features)
print("\nFeature Importance (weight):")
print(importances.sort_values(ascending=False).to_string())

XGBoost for Regression

XGBoost works equally well for regression. You only need to change the objective parameter. Here we predict house prices.

import xgboost as xgb
from sklearn.datasets import fetch_california_housing
from sklearn.model_selection import train_test_split
from sklearn.metrics import mean_absolute_error, r2_score
import numpy as np

# ── Load California housing ───────────────────────────────
data = fetch_california_housing(as_frame=True)
X, y = data.data, data.target   # y = median house value ($100k)

X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42
)
X_tr, X_val, y_tr, y_val = train_test_split(
    X_train, y_train, test_size=0.15, random_state=42
)

# ── XGBoost Regressor ─────────────────────────────────────
reg = xgb.XGBRegressor(
    objective         = 'reg:squarederror',  # MSE loss
    n_estimators      = 1000,
    learning_rate     = 0.04,
    max_depth         = 5,
    min_child_weight  = 5,
    subsample         = 0.8,
    colsample_bytree  = 0.8,
    reg_alpha         = 0.05,
    reg_lambda        = 1.5,
    gamma             = 0.1,
    tree_method       = 'hist',
    early_stopping_rounds = 40,
    eval_metric       = 'rmse',
    n_jobs            = -1,
    random_state      = 42
)

reg.fit(X_tr, y_tr, eval_set=[(X_val, y_val)], verbose=100)

# ── Evaluate ──────────────────────────────────────────────
y_pred = reg.predict(X_test)

print(f"\nTest RMSE : {np.sqrt(mean_absolute_error(y_test, y_pred)**2 + np.var(y_test - y_pred)):.4f}")
print(f"Test MAE  : {mean_absolute_error(y_test, y_pred):.4f}")
print(f"Test R²   : {r2_score(y_test, y_pred):.4f}")
print(f"Best iter : {reg.best_iteration}")

Hyperparameter Guide — The Complete Reference

XGBoost has dozens of hyperparameters. These are the ones that matter in practice, grouped by their purpose.

Hyperparameter Tuning with Optuna

Manual tuning works but Optuna's Bayesian optimisation is faster and finds better combinations. This is how competition winners tune XGBoost.

import optuna
import xgboost as xgb
from sklearn.model_selection import StratifiedKFold, cross_val_score
import numpy as np

optuna.logging.set_verbosity(optuna.logging.WARNING)  # suppress output

def objective(trial):
    params = {
        'n_estimators'     : trial.suggest_int('n_estimators', 100, 800),
        'learning_rate'    : trial.suggest_float('learning_rate', 0.01, 0.3, log=True),
        'max_depth'        : trial.suggest_int('max_depth', 2, 8),
        'min_child_weight' : trial.suggest_int('min_child_weight', 1, 10),
        'gamma'            : trial.suggest_float('gamma', 0, 5),
        'subsample'        : trial.suggest_float('subsample', 0.5, 1.0),
        'colsample_bytree' : trial.suggest_float('colsample_bytree', 0.4, 1.0),
        'reg_alpha'        : trial.suggest_float('reg_alpha', 1e-8, 10, log=True),
        'reg_lambda'       : trial.suggest_float('reg_lambda', 1e-8, 10, log=True),
        'tree_method'      : 'hist',
        'n_jobs'           : -1,
        'random_state'     : 42
    }
    model = xgb.XGBClassifier(**params)
    cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)
    scores = cross_val_score(model, X, y, cv=cv, scoring='roc_auc')
    return scores.mean()

study = optuna.create_study(direction='maximize')
study.optimize(objective, n_trials=100, timeout=300)  # 5 minutes max

print(f"Best AUC   : {study.best_value:.4f}")
print(f"Best params: {study.best_params}")

# ── Retrain final model with best params ──────────────────
best = xgb.XGBClassifier(**study.best_params)
best.fit(X, y)

Feature Importance in XGBoost — Three Types

XGBoost offers three different ways to measure feature importance. They tell different stories and you should know the difference.

import matplotlib.pyplot as plt
from xgboost import plot_importance

# ── Three importance types ─────────────────────────────────
fig, axes = plt.subplots(1, 3, figsize=(18, 5))

for ax, imp_type in zip(axes, ['weight', 'gain', 'cover']):
    plot_importance(model, importance_type=imp_type, ax=ax,
                    title=f'Importance ({imp_type})')

plt.tight_layout()
plt.savefig('xgb_importance.png', dpi=150)

# ── Or get raw values as a dict ───────────────────────────
gain_scores = model.get_booster().get_score(importance_type='gain')
for feat, score in sorted(gain_scores.items(), key=lambda x: -x[1]):
    print(f"  {feat:15s}: {score:.2f}")

# ── SHAP values — most reliable importance ────────────────
import shap
explainer   = shap.TreeExplainer(model)
shap_values = explainer.shap_values(X_test)
shap.summary_plot(shap_values, X_test, plot_type='bar')

The Full Boosting Landscape — Visual Comparison

Boosting vs Random Forest — When to Use Which

Common Mistakes & How to Avoid Them

Golden Rules