Gradient Boosting

The Story That Explains Gradient Boostingवह कहानी जो Gradient Boosting समझाती है一个解释 Gradient Boosting 的故事

Gradient Boosting is a sequential ensemble method. It builds a series of small, shallow decision trees. Each tree is trained to fix the combined error of all the trees before it. Random Forest builds trees independently and in parallel. Gradient Boosting is sequential: each tree depends on the ones before it.

Gradient Boosting एक sequential ensemble method है। यह छोटे, shallow decision trees की एक श्रृंखला बनाता है। हर tree को पिछले सभी trees की मिली-जुली error ठीक करने के लिए train किया जाता है। Random Forest trees को अलग-अलग और parallel में बनाता है। Gradient Boosting sequential है: हर tree पिछले trees पर निर्भर है।

Gradient Boosting 是一种串行的 ensemble 方法。它构建一系列又小又浅的决策树，每棵树都被训练来修正前面所有树的总误差。Random Forest 独立、并行地建树，而 Gradient Boosting 是串行的：每棵树都依赖前面的树。

Boosting vs Bagging — Two Different PhilosophiesBoosting vs Bagging — दो अलग सोचBoosting vs Bagging — 两种不同的思路

Both boosting and bagging combine many models. But how they combine them is very different.

Boosting और bagging दोनों कई models जोड़ते हैं। पर वे उन्हें कैसे जोड़ते हैं, यह बहुत अलग है।

Boosting 和 bagging 都组合多个 model，但组合方式截然不同。

The Algorithm — Step by StepAlgorithm — Step by Step算法 — 逐步讲解

Let's walk through the algorithm with a regression problem: predicting house prices.

आइए algorithm को एक regression problem से समझें: घरों की कीमत predict करना।

我们用一个回归问题来讲解算法：预测房价。

The Mathematics — What "Gradient" Actually Meansगणित — "Gradient" का असली मतलब数学 — "梯度"到底是什么意思

Gradient Boosting is gradient descent in function space. A neural network uses gradients to update its parameters. Gradient Boosting uses gradients to update the predictions directly, step by step, in the direction that lowers the loss most.

Gradient Boosting function space में gradient descent है। Neural network gradients से अपने parameters update करता है। Gradient Boosting gradients से सीधे predictions को, step दर step, उस दिशा में update करता है जहाँ loss सबसे ज़्यादा घटता है।

Gradient Boosting 是函数空间中的梯度下降。神经网络用梯度更新参数，而 Gradient Boosting 用梯度直接一步步更新预测值，朝着让 loss 下降最快的方向。

Visual Diagram — The Sequential ProcessVisual Diagram — Sequential प्रक्रिया图解 — 串行过程

Loss Functions — The Heart of FlexibilityLoss Functions — Flexibility का दिल损失函数 — 灵活性的核心

Gradient Boosting works with any differentiable loss function. The loss decides what counts as a "mistake" and how much each mistake costs.

Gradient Boosting किसी भी differentiable loss function के साथ काम करता है। Loss तय करता है कि "गलती" क्या है और हर गलती की कीमत कितनी है।

Gradient Boosting 适用于任何可微损失函数。loss 决定了什么算"错误"，以及每个错误的代价有多大。

Key Hyperparameters — The Control Panelमुख्य Hyperparameters — Control Panel关键超参数 — 控制面板

🍎 Family 1 — Ensemble Structure🍎 Family 1 — Ensemble का ढाँचा🍎 第 1 类 — 集成结构

🌵 Family 2 — Tree Structure (Complexity)🌵 Family 2 — Tree का ढाँचा (Complexity)🌵 第 2 类 — 树结构（复杂度）

🎗 Family 3 — Randomness and Early Stopping🎗 Family 3 — Randomness और Early Stopping🎗 第 3 类 — 随机性与 Early Stopping

Shrinkage — The Most Important Tuning DecisionShrinkage — Tuning का सबसे ज़रूरी फ़ैसलाShrinkage — 最重要的调参决定

Stochastic Gradient Boosting — Adding RandomnessStochastic Gradient Boosting — Randomness जोड़ना随机梯度提升 — 加入随机性

Instead of using all training rows for each tree, use a random subsample of rows (drawn without replacement). The subsample parameter controls this. Any value below 1.0 gives Stochastic Gradient Boosting. Try it in the simulator in Section 10.

हर tree के लिए सारी training rows इस्तेमाल करने के बजाय rows का random subsample (without replacement) लें। subsample parameter इसे नियंत्रित करता है। 1.0 से कम कोई भी value Stochastic Gradient Boosting देती है। इसे Section 10 के simulator में आज़माएँ।

不是每棵树都用全部训练行，而是用随机抽取的子样本（无放回）。这由 subsample 参数控制，小于 1.0 就是随机梯度提升。可以在第 10 节的模拟器里试试。

Overfitting and RegularisationOverfitting और Regularisation过拟合与正则化

Python — scikit-learn GradientBoostingPython — scikit-learn GradientBoostingPython — scikit-learn GradientBoosting

🛠️ Classification: Credit Card Fraud Detection🛠️ Classification: Credit Card Fraud Detection🛠️ 分类：信用卡欺诈检测

from sklearn.ensemble import GradientBoostingClassifier
from sklearn.model_selection import train_test_split
from sklearn.metrics import classification_report, roc_auc_score
from sklearn.datasets import make_classification

# Synthetic "credit card fraud" data: 95% legit, 5% fraud
X, y = make_classification(
    n_samples=8000, n_features=20,
    n_informative=12, n_redundant=4,
    weights=[0.95, 0.05],
    random_state=42
)
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, stratify=y, random_state=42
)

gb = GradientBoostingClassifier(
    n_estimators=500,          # upper limit; early stopping decides
    learning_rate=0.05,        # small steps, better generalisation
    max_depth=4,               # shallow trees
    min_samples_leaf=10,       # regularise leaf size
    subsample=0.75,            # stochastic GB: 75% of rows per tree
    max_features=0.5,          # 50% of features per split
    loss='log_loss',           # binary classification
    validation_fraction=0.1,   # 10% of training data for early stopping
    n_iter_no_change=20,       # stop after 20 rounds with no improvement
    tol=1e-4,
    random_state=42
)
gb.fit(X_train, y_train)

print(f"Trees used (early stopping): {gb.n_estimators_}")
print(f"Test ROC-AUC: {roc_auc_score(y_test, gb.predict_proba(X_test)[:, 1]):.4f}\n")
print(classification_report(y_test, gb.predict(X_test), digits=4))

from sklearn.metrics import precision_score, recall_score

# `gb`, `X_test`, `y_test` come from the fraud example above
proba = gb.predict_proba(X_test)[:, 1]

print(f"{'Threshold':>9} {'Precision':>10} {'Recall':>8} {'Frauds caught':>14}")
for t in [0.5, 0.3, 0.2, 0.1, 0.05]:
    pred = (proba >= t).astype(int)
    caught = int(((pred == 1) & (y_test == 1)).sum())
    print(f"{t:>9} {precision_score(y_test, pred):>10.3f} "
          f"{recall_score(y_test, pred):>8.3f} {caught:>7} / {int(y_test.sum())}")

At a 0.1 threshold the model catches 65 of 87 frauds instead of 40. Pick the threshold from the real cost of a missed fraud versus a false alarm, and choose it on validation data, not the test set.

0.1 threshold पर model 87 में से 40 के बजाय 65 frauds पकड़ता है। Threshold को छूटे fraud बनाम झूठे alarm की असली कीमत से चुनें, और इसे validation data पर चुनें, test set पर नहीं।

阈值为 0.1 时，model 抓到 87 个欺诈中的 65 个，而不是 40 个。阈值应根据漏掉欺诈与误报的真实代价来定，并在验证集上选择，而不是测试集。

📈 Regression: Predicting House Prices📈 Regression: घरों की कीमत Predict करना📈 回归：预测房价

from sklearn.ensemble import GradientBoostingRegressor
from sklearn.datasets import fetch_california_housing
from sklearn.model_selection import train_test_split
from sklearn.metrics import mean_absolute_error, r2_score

housing = fetch_california_housing()
X, y = housing.data, housing.target        # y = median house value in $100,000
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42
)

gb_reg = GradientBoostingRegressor(
    n_estimators=2000,
    learning_rate=0.05,
    max_depth=4,
    subsample=0.8,
    max_features='sqrt',
    loss='huber',              # robust to outlier prices
    alpha=0.9,                 # Huber switches to linear beyond the 90th percentile of |residual|
    validation_fraction=0.1,
    n_iter_no_change=25,
    random_state=42
)
gb_reg.fit(X_train, y_train)
y_pred = gb_reg.predict(X_test)

print(f"R² Score : {r2_score(y_test, y_pred):.4f}")
print(f"MAE      : ${mean_absolute_error(y_test, y_pred) * 100_000:,.0f}")
print(f"Trees    : {gb_reg.n_estimators_} (early stopped)")

LightGBM — Faster, Scalable Gradient BoostingLightGBM — तेज़, Scalable Gradient BoostingLightGBM — 更快、可扩展的梯度提升

LightGBM is Microsoft's high-speed implementation. The residual-fitting logic is the same, but two engineering ideas make it much faster on large data: histogram-based splits and leaf-wise tree growth.

LightGBM Microsoft का तेज़ implementation है। Residual-fitting का तर्क वही है, पर दो engineering ideas इसे बड़े data पर बहुत तेज़ बनाते हैं: histogram-based splits और leaf-wise tree growth.

LightGBM 是微软的高速实现。拟合残差的逻辑相同，但两个工程上的创新让它在大数据上快得多：基于直方图的分裂和按叶生长。

import lightgbm as lgb
from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split
from sklearn.metrics import roc_auc_score, classification_report

# Same fraud data as before, split into train / validation / test
X, y = make_classification(n_samples=8000, n_features=20, n_informative=12,
                           n_redundant=4, weights=[0.95, 0.05], random_state=42)
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, stratify=y, random_state=42)
X_tr, X_val, y_tr, y_val = train_test_split(
    X_train, y_train, test_size=0.15, stratify=y_train, random_state=42)

train_data = lgb.Dataset(X_tr,  label=y_tr)
val_data   = lgb.Dataset(X_val, label=y_val, reference=train_data)

params = {
    'objective':         'binary',
    'metric':            'binary_logloss',
    'learning_rate':     0.05,
    'num_leaves':        31,      # main complexity control in LightGBM
    'max_depth':         -1,      # -1 = no limit; num_leaves controls size
    'min_child_samples': 20,
    'feature_fraction':  0.7,
    'bagging_fraction':  0.8,
    'bagging_freq':      5,
    'lambda_l1':         0.1,
    'lambda_l2':         1.0,
    'verbosity':         -1,
    'seed':              42
}
callbacks = [lgb.early_stopping(stopping_rounds=50, verbose=False),
             lgb.log_evaluation(period=100)]
model = lgb.train(params, train_data, num_boost_round=2000,
                  valid_sets=[val_data], callbacks=callbacks)

y_prob = model.predict(X_test, num_iteration=model.best_iteration)
print(f"\nBest iteration : {model.best_iteration}")
print(f"Test ROC-AUC   : {roc_auc_score(y_test, y_prob):.4f}\n")
print(classification_report(y_test, (y_prob > 0.5).astype(int), digits=4))

LightGBM matches the scikit-learn model (AUC 0.946 vs 0.943) and stops after 152 rounds. Note the three-way split: early stopping uses the validation set, and the test set is used only once at the end.

LightGBM, scikit-learn model के बराबर है (AUC 0.946 बनाम 0.943) और 152 rounds के बाद रुकता है। तीन हिस्सों वाला split देखें: early stopping validation set इस्तेमाल करता है, और test set सिर्फ़ आख़िर में एक बार।

LightGBM 与 scikit-learn model 不相上下（AUC 0.946 vs 0.943），在 152 轮后停止。注意三段式划分：early stopping 用验证集，测试集只在最后用一次。

⏱️ Speed Check: 200,000 Rows⏱️ Speed जाँच: 200,000 Rows⏱️ 速度测试：20 万行

import time
import lightgbm as lgb
from sklearn.datasets import make_classification
from sklearn.ensemble import GradientBoostingClassifier, HistGradientBoostingClassifier
from sklearn.model_selection import train_test_split
from sklearn.metrics import roc_auc_score

X, y = make_classification(n_samples=200_000, n_features=20,
                           n_informative=12, random_state=42)
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.2, random_state=42)

models = {
    'GradientBoosting (exact)':  GradientBoostingClassifier(n_estimators=200, max_depth=3, random_state=42),
    'HistGradientBoosting':      HistGradientBoostingClassifier(max_iter=200, max_depth=3, early_stopping=False, random_state=42),
    'LightGBM':                  lgb.LGBMClassifier(n_estimators=200, max_depth=3, num_leaves=8, verbosity=-1, random_state=42),
}
print(f"{'Model':<26} {'Train time':>10} {'Test AUC':>9}")
for name, m in models.items():
    t0 = time.perf_counter()
    m.fit(X_tr, y_tr)
    secs = time.perf_counter() - t0
    auc = roc_auc_score(y_te, m.predict_proba(X_te)[:, 1])
    print(f"{name:<26} {secs:>9.1f}s {auc:>9.4f}")

Feature Importance in Gradient BoostingGradient Boosting में Feature Importance梯度提升中的特征重要性

import numpy as np
from sklearn.metrics import roc_auc_score

# `model`, `X_test`, `y_test` come from the LightGBM example above
best = model.best_iteration
split = model.feature_importance(importance_type='split')   # how often used
gain  = model.feature_importance(importance_type='gain')    # how much loss it removed

# Permutation importance: shuffle one column, see how much test AUC drops
base = roc_auc_score(y_test, model.predict(X_test, num_iteration=best))
rng = np.random.default_rng(42)
perm = []
for j in range(X_test.shape[1]):
    X_shuffled = X_test.copy()
    X_shuffled[:, j] = rng.permutation(X_shuffled[:, j])
    perm.append(base - roc_auc_score(y_test, model.predict(X_shuffled, num_iteration=best)))

print("Top 5 features by each method:")
for name, scores in [('split', split), ('gain', gain), ('permutation', np.array(perm))]:
    top = np.argsort(scores)[::-1][:5]
    print(f"  {name:<12}: " + ", ".join(f"f{i}" for i in top))
print(f"\nShuffling f{int(np.argmax(perm))} drops test AUC by {max(perm):.4f}")

The three methods disagree. Split count ranks f5 second, but by permutation on test data it ranks only 12th of 20. Permutation importance ranks f14 first: shuffling it drops test AUC by 0.04. When the ranking matters for a decision, trust held-out methods (permutation or SHAP).

तीनों methods सहमत नहीं हैं। Split count f5 को दूसरे स्थान पर रखता है, पर test data पर permutation से वह 20 में से सिर्फ़ 12वें स्थान पर है। Permutation importance f14 को पहले रखता है: उसे shuffle करने से test AUC 0.04 गिरता है। जब ranking किसी फ़ैसले के लिए मायने रखे, तो held-out methods (permutation या SHAP) पर भरोसा करें।

三种方法结论不一致。分裂次数把 f5 排第二，但按测试数据上的 permutation，它在 20 个特征中只排第 12。Permutation 重要性把 f14 排第一：打乱它会让测试 AUC 下降 0.04。当排名关系到决策时，请相信基于留出数据的方法（permutation 或 SHAP）。

Gradient Boosting vs Random Forest — Full ComparisonGradient Boosting vs Random Forest — पूरी तुलनाGradient Boosting vs Random Forest — 全面对比

Golden RulesGolden Rules黄金法则