Boosting & XGBoost

The Story That Explains Boostingवह कहानी जो Boosting समझाती है一个解释 Boosting 的故事

Boosting is a family of ensemble learning methods. The idea: combine many weak learners (models only a little better than guessing) into one strong learner. Each weak learner fixes what the earlier ones could not. The three most important members are:

Boosting, ensemble learning methods का एक परिवार है। विचार: कई weak learners (जो अंदाज़े से बस थोड़े बेहतर हों) को जोड़कर एक strong learner बनाना। हर weak learner वह ठीक करता है जो पहले वाले नहीं कर पाए। इसके तीन सबसे ज़रूरी सदस्य हैं:

Boosting 是一类 ensemble learning 方法。思路：把许多 weak learner（只比瞎猜好一点的 model）组合成一个 strong learner。每个 weak learner 修正前面的 learner 没能解决的问题。三个最重要的成员是：

AdaBoost — Where It All StartedAdaBoost — जहाँ से सब शुरू हुआAdaBoost — 一切的起点

AdaBoost (Adaptive Boosting) was introduced by Freund & Schapire in the mid-1990s. Their work won the 2003 Gödel Prize. It is the simplest boosting method, and once you understand it, everything after it is easier.

AdaBoost (Adaptive Boosting) को Freund & Schapire ने 1990 के दशक के मध्य में पेश किया। उनके काम को 2003 का Gödel Prize मिला। यह सबसे सरल boosting method है, और इसे समझने के बाद आगे की हर चीज़ आसान हो जाती है।

AdaBoost（Adaptive Boosting）由 Freund 和 Schapire 在 1990 年代中期提出，他们的工作获得了 2003 年哥德尔奖。它是最简单的 boosting 方法，理解了它，后面的内容都会更容易。

The AdaBoost Algorithm — Step by StepAdaBoost Algorithm — Step by StepAdaBoost 算法 — 逐步讲解

Gradient Boosting — The GeneralisationGradient Boosting — सामान्य रूपGradient Boosting — 推广形式

Gradient Boosting (Friedman, 1999; published 2001) took the idea of AdaBoost and turned it into gradient descent in function space. Instead of re-weighting samples, each new tree directly predicts the residual errors (more precisely, the negative gradients of the loss) of the current model.

Gradient Boosting (Friedman, 1999; प्रकाशित 2001) ने AdaBoost के विचार को function space में gradient descent बना दिया। Samples का weight बदलने के बजाय हर नया tree सीधे मौजूदा model की residual errors (सटीक रूप से, loss के negative gradients) predict करता है।

Gradient Boosting（Friedman，1999 年提出，2001 年发表）把 AdaBoost 的思想变成了函数空间中的梯度下降。它不再重新加权样本，而是让每棵新树直接预测当前 model 的残差（更准确地说，是 loss 的负梯度）。

Gradient Boosting — The AlgorithmGradient Boosting — AlgorithmGradient Boosting — 算法

Gradient Boosting From Scratch — 15 LinesGradient Boosting Scratch से — 15 Lines从零实现 Gradient Boosting — 15 行

The four formulas above fit in a short loop. The scikit-learn version with the same settings gives exactly the same training error, so this really is the whole algorithm (for MSE loss).

ऊपर के चार formulas एक छोटे loop में आ जाते हैं। उन्हीं settings वाला scikit-learn version बिल्कुल वही training error देता है, इसलिए (MSE loss के लिए) यही पूरा algorithm है।

上面四个公式可以写成一个很短的循环。相同设置下的 scikit-learn 版本给出完全相同的训练误差，所以（对 MSE loss 来说）这就是整个算法。

import numpy as np
from sklearn.tree import DecisionTreeRegressor
from sklearn.ensemble import GradientBoostingRegressor

# Toy data: a noisy sine wave
rng = np.random.default_rng(42)
X = np.sort(rng.uniform(0, 10, 200)).reshape(-1, 1)
y = 3 * np.sin(X).ravel() + rng.normal(0, 0.5, 200)

eta, n_rounds = 0.1, 100
F = np.full_like(y, y.mean())          # Step 1: F0 = mean of y
trees = []

for m in range(1, n_rounds + 1):
    residuals = y - F                  # Step 2: residuals (negative gradient of MSE)
    tree = DecisionTreeRegressor(max_depth=2).fit(X, residuals)   # Step 3
    F += eta * tree.predict(X)         # Step 4: F_m = F_(m-1) + eta * h_m
    trees.append(tree)
    if m in (1, 5, 20, 50, 100):
        print(f"Round {m:3d}: train MSE = {np.mean((y - F) ** 2):.4f}")

# Same settings in scikit-learn give the same answer
sk = GradientBoostingRegressor(n_estimators=100, learning_rate=0.1,
                               max_depth=2, random_state=0).fit(X, y)
print(f"\nOur model      : train MSE = {np.mean((y - F) ** 2):.4f}")
print(f"scikit-learn   : train MSE = {np.mean((y - sk.predict(X)) ** 2):.4f}")

XGBoost — Extreme Gradient BoostingXGBoost — Extreme Gradient BoostingXGBoost — 极端梯度提升

What Makes XGBoost DifferentXGBoost को क्या अलग बनाता हैXGBoost 有何不同

The XGBoost Objective FunctionXGBoost Objective FunctionXGBoost 目标函数

Python Implementation — AdaBoostPython Implementation — AdaBoostPython 实现 — AdaBoost

Let's start with AdaBoost on a classic yes/no problem: will a bank customer default on a loan? (The data is synthetic.)

AdaBoost से एक classic हाँ/नहीं वाले सवाल पर शुरू करें: क्या bank customer loan default करेगा? (Data synthetic है।)

先用 AdaBoost 解决一个经典的是/否问题：银行客户会不会贷款违约？（数据是合成的。）

import numpy as np
from sklearn.ensemble import AdaBoostClassifier
from sklearn.tree import DecisionTreeClassifier
from sklearn.model_selection import train_test_split, cross_val_score
from sklearn.metrics import classification_report, roc_auc_score
from sklearn.datasets import make_classification

# ── Synthetic "loan default" dataset ──────────────────────
X, y = make_classification(
    n_samples=5000, n_features=15, n_informative=10,
    n_redundant=3, class_sep=0.8, random_state=42
)
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42, stratify=y
)

# ── AdaBoost with decision stumps (depth = 1) ─────────────
# Note: do NOT pass algorithm='SAMME'. That parameter was removed
# from recent scikit-learn; SAMME is now the only algorithm.
ada = AdaBoostClassifier(
    estimator=DecisionTreeClassifier(max_depth=1),
    n_estimators=200,        # number of stumps / boosting rounds
    learning_rate=0.5,       # shrinks each stump's vote
    random_state=42
)

# ── Cross-validation ──────────────────────────────────────
cv_auc = cross_val_score(ada, X_train, y_train, cv=5, scoring='roc_auc')
print(f"CV ROC-AUC  : {cv_auc.mean():.4f} ± {cv_auc.std():.4f}")

# ── Fit and evaluate ──────────────────────────────────────
ada.fit(X_train, y_train)
y_pred  = ada.predict(X_test)
y_proba = ada.predict_proba(X_test)[:, 1]
print(f"Test ROC-AUC: {roc_auc_score(y_test, y_proba):.4f}\n")
print(classification_report(y_test, y_pred))

# ── Look at the stump weights (alpha) ─────────────────────
print("Top 5 stump weights:")
for i in np.argsort(ada.estimator_weights_)[::-1][:5]:
    print(f"  Stump {i:3d}: weight = {ada.estimator_weights_[i]:.4f}")

Python Implementation — XGBoost End-to-EndPython Implementation — XGBoost शुरू से अंत तकPython 实现 — XGBoost 端到端

Now the real workhorse: a full XGBoost pipeline on the classic Titanic survival data (Kaggle train.csv, 891 passengers), with feature engineering, early stopping and annotated hyperparameters.

अब असली workhorse: classic Titanic survival data (Kaggle train.csv, 891 passengers) पर पूरी XGBoost pipeline, feature engineering, early stopping और समझाए गए hyperparameters के साथ।

现在来看真正的主力：在经典的 Titanic 生存数据（Kaggle train.csv，891 名乘客）上搭建完整的 XGBoost 流程，包括特征工程、early stopping 和带注释的超参数。

import xgboost as xgb
import pandas as pd
from sklearn.model_selection import train_test_split, StratifiedKFold, cross_val_score
from sklearn.preprocessing import LabelEncoder

# ── Load data (Kaggle Titanic train.csv) ──────────────────
df = pd.read_csv('titanic.csv')

# ── Feature engineering ───────────────────────────────────
df['Title'] = df['Name'].str.extract(r' ([A-Za-z]+)\.', expand=False)
df['Title'] = df['Title'].replace({
    'Mlle': 'Miss', 'Ms': 'Miss', 'Mme': 'Mrs',
    'Lady': 'Rare', 'Countess': 'Rare', 'Capt': 'Rare', 'Col': 'Rare',
    'Don': 'Rare', 'Dr': 'Rare', 'Major': 'Rare', 'Rev': 'Rare',
    'Sir': 'Rare', 'Jonkheer': 'Rare', 'Dona': 'Rare'
})
df['FamilySize'] = df['SibSp'] + df['Parch'] + 1
df['IsAlone']    = (df['FamilySize'] == 1).astype(int)

# pandas 3: assign back. df['Age'].fillna(..., inplace=True) does NOTHING.
# (XGBoost can also handle NaN itself; we fill here only for Embarked/Fare.)
df['Embarked'] = df['Embarked'].fillna(df['Embarked'].mode()[0])
df['Fare']     = df['Fare'].fillna(df['Fare'].median())

# Encode text columns as integers
for col in ['Sex', 'Embarked', 'Title']:
    df[col] = LabelEncoder().fit_transform(df[col].astype(str))

features = ['Pclass', 'Sex', 'Age', 'SibSp', 'Parch',
            'Fare', 'Embarked', 'Title', 'FamilySize', 'IsAlone']
X, y = df[features], df['Survived']     # Age keeps its NaNs on purpose

# ── XGBoost model — annotated hyperparameters ─────────────
model = xgb.XGBClassifier(
    # Boosting structure
    n_estimators     = 1000,     # upper limit; early stopping picks the real number
    learning_rate    = 0.05,     # eta: shrinks each tree's contribution
    max_depth        = 4,        # tree depth: 3-6 is typical
    # Regularisation
    reg_alpha        = 0.1,      # L1 on leaf weights
    reg_lambda       = 1.0,      # L2 on leaf weights
    gamma            = 0.05,     # minimum gain needed to split
    min_child_weight = 3,        # minimum Hessian sum in a leaf
    # Randomisation (like Random Forest)
    subsample        = 0.8,      # rows per tree
    colsample_bytree = 0.7,      # columns per tree
    # Performance
    tree_method      = 'hist',   # fast histogram splits (the default in XGBoost 2+)
    device           = 'cpu',    # 'cuda' for GPU
    n_jobs           = -1,
    random_state     = 42,
    eval_metric      = 'logloss',  # smooth metric: better for early stopping than AUC
    early_stopping_rounds = 50   # stop after 50 rounds with no improvement
)

# ── Hold out a validation set for early stopping ──────────
X_tr, X_val, y_tr, y_val = train_test_split(
    X, y, test_size=0.2, random_state=42, stratify=y
)
model.fit(X_tr, y_tr, eval_set=[(X_val, y_val)], verbose=50)

print(f"\nBest iteration : {model.best_iteration}")
print(f"Best val loss  : {model.best_score:.4f}")

# ── Cross-validated AUC with the chosen number of trees ───
cv_model = xgb.XGBClassifier(
    n_estimators=model.best_iteration + 1,   # best_iteration is 0-based
    learning_rate=0.05, max_depth=4, subsample=0.8,
    colsample_bytree=0.7, tree_method='hist', n_jobs=-1, random_state=42
)
cv = StratifiedKFold(5, shuffle=True, random_state=42)
cv_auc = cross_val_score(cv_model, X, y, cv=cv, scoring='roc_auc')
print(f"CV ROC-AUC     : {cv_auc.mean():.4f} ± {cv_auc.std():.4f}")

# ── Feature importance (the default type is 'gain') ───────
imp = pd.Series(model.feature_importances_, index=features)
print("\nFeature importance (gain):")
print(imp.sort_values(ascending=False).round(4).to_string())

XGBoost for RegressionRegression के लिए XGBoostXGBoost 回归

XGBoost works just as well for regression. You mainly change the objective. Here we predict house prices in California.

XGBoost regression के लिए भी उतना ही अच्छा है। मुख्य रूप से आप objective बदलते हैं। यहाँ हम California में घरों की कीमत predict करते हैं।

XGBoost 做回归同样出色，主要只需改 objective。这里我们预测加州的房价。

import numpy as np
import xgboost as xgb
from sklearn.datasets import fetch_california_housing
from sklearn.model_selection import train_test_split
from sklearn.metrics import root_mean_squared_error, mean_absolute_error, r2_score

# ── Load California housing ───────────────────────────────
data = fetch_california_housing(as_frame=True)
X, y = data.data, data.target          # y = median house value in $100,000

X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42
)
X_tr, X_val, y_tr, y_val = train_test_split(
    X_train, y_train, test_size=0.15, random_state=42
)

# ── XGBoost regressor ─────────────────────────────────────
reg = xgb.XGBRegressor(
    objective        = 'reg:squarederror',   # MSE loss
    n_estimators     = 3000,
    learning_rate    = 0.04,
    max_depth        = 5,
    min_child_weight = 5,
    subsample        = 0.8,
    colsample_bytree = 0.8,
    reg_alpha        = 0.05,
    reg_lambda       = 1.5,
    gamma            = 0.1,
    tree_method      = 'hist',
    eval_metric      = 'rmse',
    early_stopping_rounds = 50,
    n_jobs           = -1,
    random_state     = 42
)
reg.fit(X_tr, y_tr, eval_set=[(X_val, y_val)], verbose=200)

# ── Evaluate on the untouched test set ────────────────────
y_pred = reg.predict(X_test)
print(f"\nTest RMSE : {root_mean_squared_error(y_test, y_pred):.4f}")
print(f"Test MAE  : {mean_absolute_error(y_test, y_pred):.4f}")
print(f"Test R²   : {r2_score(y_test, y_pred):.4f}")
print(f"Best iter : {reg.best_iteration}")

Hyperparameter Guide — The Complete ReferenceHyperparameter Guide — पूरा Reference超参数指南 — 完整参考

XGBoost has dozens of hyperparameters. These are the ones that matter in practice. Defaults are for XGBoost 2.x/3.x.

XGBoost में दर्जनों hyperparameters हैं। ये वे हैं जो असल में मायने रखते हैं। Defaults XGBoost 2.x/3.x के हैं।

XGBoost 有几十个超参数，下面是实践中真正重要的。默认值适用于 XGBoost 2.x/3.x。

Hyperparameter Tuning with OptunaOptuna से Hyperparameter Tuning用 Optuna 调超参数

Manual tuning works, but Optuna's smart (Bayesian-style) search is faster and often finds better combinations. Here it searches 9 hyperparameters on the Titanic data from Section 06.

Manual tuning काम करती है, पर Optuna की smart (Bayesian-style) search तेज़ है और अक्सर बेहतर combinations खोजती है। यहाँ यह Section 06 के Titanic data पर 9 hyperparameters खोजती है।

手动调参可行，但 Optuna 的智能（贝叶斯式）搜索更快，常常能找到更好的组合。这里它在第 06 节的 Titanic 数据上搜索 9 个超参数。

import optuna
import xgboost as xgb
from sklearn.model_selection import StratifiedKFold, cross_val_score

optuna.logging.set_verbosity(optuna.logging.WARNING)   # quiet output

# X, y: the Titanic features from the previous section
def objective(trial):
    params = {
        'n_estimators'    : trial.suggest_int('n_estimators', 100, 800),
        'learning_rate'   : trial.suggest_float('learning_rate', 0.01, 0.3, log=True),
        'max_depth'       : trial.suggest_int('max_depth', 2, 8),
        'min_child_weight': trial.suggest_int('min_child_weight', 1, 10),
        'gamma'           : trial.suggest_float('gamma', 0, 5),
        'subsample'       : trial.suggest_float('subsample', 0.5, 1.0),
        'colsample_bytree': trial.suggest_float('colsample_bytree', 0.4, 1.0),
        'reg_alpha'       : trial.suggest_float('reg_alpha', 1e-8, 10, log=True),
        'reg_lambda'      : trial.suggest_float('reg_lambda', 1e-8, 10, log=True),
    }
    model = xgb.XGBClassifier(**params, tree_method='hist', n_jobs=-1, random_state=42)
    cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)
    return cross_val_score(model, X, y, cv=cv, scoring='roc_auc').mean()

study = optuna.create_study(direction='maximize',
                            sampler=optuna.samplers.TPESampler(seed=42))
study.optimize(objective, n_trials=100, timeout=300)   # 5 minutes max

print(f"Trials run : {len(study.trials)}")
print(f"Best AUC   : {study.best_value:.4f}")
print("Best params:")
for k, v in study.best_params.items():
    print(f"  {k:<17}: {v:.4g}" if isinstance(v, float) else f"  {k:<17}: {v}")

# ── Retrain the final model with the best params ──────────
best = xgb.XGBClassifier(**study.best_params, tree_method='hist',
                         n_jobs=-1, random_state=42)
best.fit(X, y)

After 100 trials, the best CV AUC is 0.889, up from 0.881 with our hand-picked settings. Notice the pattern Optuna found: a very low learning rate (0.015) with many trees (479). That matches the advice in the table above.

100 trials के बाद best CV AUC 0.889 है, हमारी हाथ से चुनी settings के 0.881 से ज़्यादा। Optuna को मिला pattern देखें: बहुत कम learning rate (0.015) और कई trees (479)। यह ऊपर की table की सलाह से मेल खाता है।

100 次试验后，最佳 CV AUC 为 0.889，高于我们手动选择参数时的 0.881。注意 Optuna 找到的规律：很低的学习率（0.015）加上很多树（479 棵），这和上表的建议一致。

Feature Importance in XGBoost — Three TypesXGBoost में Feature Importance — तीन प्रकारXGBoost 特征重要性 — 三种类型

XGBoost can measure feature importance in three ways. They tell different stories, and on real data they can disagree a lot.

XGBoost feature importance को तीन तरीकों से माप सकता है। वे अलग कहानियाँ बताते हैं, और असली data पर उनमें बहुत अंतर हो सकता है।

XGBoost 可以用三种方式衡量特征重要性。它们讲述的故事不同，在真实数据上可能差别很大。

import matplotlib.pyplot as plt
from xgboost import plot_importance
import shap

# `model` and `X_val` come from the Titanic example above
booster = model.get_booster()

# ── The three built-in importance types ───────────────────
for imp_type in ['weight', 'gain', 'cover']:
    scores = booster.get_score(importance_type=imp_type)
    top3 = sorted(scores.items(), key=lambda kv: -kv[1])[:3]
    print(f"{imp_type:<7}: " + ", ".join(f"{f} ({v:.1f})" for f, v in top3))

fig, axes = plt.subplots(1, 3, figsize=(18, 5))
for ax, imp_type in zip(axes, ['weight', 'gain', 'cover']):
    plot_importance(model, importance_type=imp_type, ax=ax,
                    title=f'Importance ({imp_type})')
plt.tight_layout()
plt.savefig('xgb_importance.png', dpi=150)

# ── SHAP values: the most reliable importance ─────────────
explainer   = shap.TreeExplainer(model)
shap_values = explainer.shap_values(X_val)
mean_abs = abs(shap_values).mean(axis=0)
top3 = sorted(zip(X_val.columns, mean_abs), key=lambda kv: -kv[1])[:3]
print("SHAP   : " + ", ".join(f"{f} ({v:.3f})" for f, v in top3))
shap.summary_plot(shap_values, X_val, plot_type='bar', show=False)

The Full Boosting LandscapeBoosting का पूरा नक्शाBoosting 全景图

Boosting vs Random Forest — When to Use WhichBoosting vs Random Forest — कब कौन-साBoosting vs Random Forest — 何时用哪个

Common Mistakes & How to Avoid Themआम गलतियाँ और उनसे कैसे बचें常见错误及避免方法

Golden RulesGolden Rules黄金法则