XGBoost Explained

The Story That Explains XGBoostवह कहानी जो XGBoost समझाती है一个解释 XGBoost 的故事

XGBoost was created by Tianqi Chen in 2014 and led Kaggle leaderboards for years. It is still one of the strongest tools for tabular data. It combines gradient boosting with regularisation, missing-value handling and fast parallel computation.

XGBoost को Tianqi Chen ने 2014 में बनाया और यह सालों तक Kaggle leaderboards पर आगे रहा। Tabular data के लिए यह अब भी सबसे मज़बूत tools में से एक है। यह gradient boosting को regularisation, missing values संभालने और तेज़ parallel computation के साथ जोड़ता है।

XGBoost 由陈天奇于 2014 年创建，多年来一直领跑 Kaggle 排行榜，至今仍是处理表格数据最强的工具之一。它把梯度提升与正则化、缺失值处理和快速的并行计算结合在一起。

XGBoost in Pictures — Understand It Before the Mathsतस्वीरों में XGBoost — गणित से पहले समझें图解 XGBoost — 先理解，再学数学

Before any formula, here is the whole idea in four pictures. If you understand these, the rest of the page is just details.

किसी भी formula से पहले, पूरा विचार चार तस्वीरों में। अगर ये समझ आ गईं, तो बाकी page सिर्फ़ details है।

在任何公式之前，先用四张图讲清楚整个思想。看懂了这些，页面其余部分只是细节。

Picture 3 — The Knobs (Hyperparameters) in Everyday Wordsतस्वीर 3 — Knobs (Hyperparameters) रोज़ की भाषा में图 3 — 用日常语言解释旋钮（超参数）

How Gradient Boosting Works — Step by StepGradient Boosting कैसे काम करता है — Step by Step梯度提升如何工作 — 逐步讲解

Before XGBoost's engineering, you need its mathematical heart: gradient boosting. Each tree is built to lower a loss function, using its gradient (the direction of fastest improvement).

XGBoost की engineering से पहले उसका गणितीय दिल समझें: gradient boosting. हर tree एक loss function को घटाने के लिए बनता है, उसके gradient (सबसे तेज़ सुधार की दिशा) की मदद से।

在了解 XGBoost 的工程实现之前，先要理解它的数学核心：梯度提升。每棵树都借助损失函数的梯度（改进最快的方向）来降低损失。

The XGBoost Objective Function — Math Explained SimplyXGBoost Objective Function — आसान गणितXGBoost 目标函数 — 简单讲数学

At each boosting round, XGBoost minimises this objective:

हर boosting round में XGBoost यह objective कम करता है:

每一轮提升中，XGBoost 最小化以下目标：

What Makes XGBoost Special — Six Engineering IdeasXGBoost को क्या ख़ास बनाता है — छह Engineering IdeasXGBoost 的特别之处 — 六个工程创新

Python Implementation — Classification & RegressionPython Implementation — Classification और RegressionPython 实现 — 分类与回归

InstallationInstallation安装

pip install xgboost scikit-learn pandas numpy

Classification — Predicting Customer ChurnClassification — Customer Churn Predict करना分类 — 预测客户流失

import xgboost as xgb
from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split
from sklearn.metrics import accuracy_score, classification_report

# Simulated churn dataset: 8,000 customers, 20 features
X, y = make_classification(
    n_samples=8000, n_features=20,
    n_informative=12, n_redundant=4,
    random_state=42
)
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42
)

# XGBoost classifier with sensible starting values
model = xgb.XGBClassifier(
    n_estimators=300,        # number of boosting rounds
    learning_rate=0.05,      # eta: shrinkage per round
    max_depth=6,             # maximum tree depth
    subsample=0.8,           # row sampling per tree
    colsample_bytree=0.8,    # feature sampling per tree
    gamma=0,                 # minimum split gain
    reg_alpha=0,             # L1 regularisation
    reg_lambda=1,            # L2 regularisation
    eval_metric='logloss',
    random_state=42
)
# Note: do NOT pass use_label_encoder — it was removed in XGBoost 2.0.

model.fit(
    X_train, y_train,
    eval_set=[(X_test, y_test)],   # only to watch the loss; no early stopping here
    verbose=100
)

y_pred = model.predict(X_test)
print(f"\nAccuracy: {accuracy_score(y_test, y_pred):.4f}\n")
print(classification_report(y_test, y_pred))

Regression — California Housing PricesRegression — California के घरों की कीमतें回归 — 加州房价

import xgboost as xgb
from sklearn.datasets import fetch_california_housing
from sklearn.model_selection import train_test_split
from sklearn.metrics import r2_score, mean_absolute_error

housing = fetch_california_housing()
X_h, y_h = housing.data, housing.target         # y = median house value in $100,000
Xh_train, Xh_test, yh_train, yh_test = train_test_split(
    X_h, y_h, test_size=0.2, random_state=42
)

reg = xgb.XGBRegressor(
    n_estimators=500,
    learning_rate=0.03,
    max_depth=5,
    subsample=0.75,
    colsample_bytree=0.75,
    reg_lambda=1.5,
    random_state=42
)
reg.fit(Xh_train, yh_train)

y_pred = reg.predict(Xh_test)
print(f"R² Score: {r2_score(yh_test, y_pred):.4f}")
print(f"MAE:      ${mean_absolute_error(yh_test, y_pred) * 100_000:,.0f}")

Key Hyperparameters — Complete Referenceमुख्य Hyperparameters — पूरा Reference关键超参数 — 完整参考

XGBoost's hyperparameters fall into four groups. Defaults are for XGBoost 2.x/3.x.

XGBoost के hyperparameters चार groups में आते हैं। Defaults XGBoost 2.x/3.x के हैं।

XGBoost 的超参数分为四组。默认值适用于 XGBoost 2.x/3.x。

Group 1 — Boosting ControlGroup 1 — Boosting नियंत्रण第 1 组 — 提升控制

Group 2 — Tree StructureGroup 2 — Tree का ढाँचा第 2 组 — 树结构

Group 3 — Sampling (Stochastic Boosting)Group 3 — Sampling (Stochastic Boosting)第 3 组 — 抽样（随机提升）

Group 4 — RegularisationGroup 4 — Regularisation第 4 组 — 正则化

Early Stopping — Finding the Right Number of TreesEarly Stopping — Trees की सही संख्या खोजनाEarly Stopping — 找到合适的树数量

import xgboost as xgb
from sklearn.model_selection import train_test_split

# Split a validation set off the TRAINING data (never use the test set here)
X_tr, X_val, y_tr, y_val = train_test_split(
    X_train, y_train, test_size=0.15, random_state=42
)

model = xgb.XGBClassifier(
    n_estimators=2000,          # set high: early stopping finds the real number
    learning_rate=0.05,
    max_depth=6,
    subsample=0.8,
    colsample_bytree=0.8,
    eval_metric='logloss',
    early_stopping_rounds=50,   # stop after 50 rounds with no improvement
    random_state=42
)
model.fit(X_tr, y_tr, eval_set=[(X_val, y_val)], verbose=False)

print(f"Best round:       {model.best_iteration}")
print(f"Best val logloss: {model.best_score:.5f}")
print(f"Trees built:      {model.get_booster().num_boosted_rounds()} of 2000")
# For a final model, use best_iteration + 1 trees (best_iteration counts from 0)

The Four Tuning Strategies — Overviewचार Tuning Strategies — एक नज़र四种调参策略 — 概览

Hyperparameter tuning means finding the combination of settings that gives the best model. There are four main strategies. They trade off speed, coverage and compute cost very differently.

Hyperparameter tuning का मतलब है settings का वह मेल खोजना जो सबसे अच्छा model दे। चार मुख्य strategies हैं। ये speed, coverage और compute cost के बीच बहुत अलग-अलग संतुलन बनाती हैं।

超参数调优就是找到能得到最佳 model 的参数组合。主要有四种策略，它们在速度、覆盖面和计算成本之间的取舍非常不同。

Strategy 1 — Grid SearchStrategy 1 — Grid Search策略 1 — 网格搜索

Grid Search trains a model for every combination of the values you list. 3 parameters with 5 values each = 5³ = 125 models. 6 parameters = 5⁶ = 15,625 models, and with 5-fold CV that is 78,125 training runs.

Grid Search आपकी दी हुई values की हर combination के लिए एक model train करता है। 5-5 values वाले 3 parameters = 5³ = 125 models. 6 parameters = 5⁶ = 15,625 models, और 5-fold CV के साथ यह 78,125 training runs है।

网格搜索会为你列出的值的每一种组合训练一个 model。3 个参数各 5 个值 = 5³ = 125 个 model。6 个参数 = 5⁶ = 15,625 个 model，加上 5 折交叉验证就是 78,125 次训练。

import time
import pandas as pd
import xgboost as xgb
from sklearn.model_selection import GridSearchCV

# Keep the grid SMALL: 3 params x 3 values = 27 combos x 5 folds = 135 fits
param_grid = {
    'max_depth':     [3, 5, 7],
    'learning_rate': [0.01, 0.05, 0.1],
    'n_estimators':  [100, 300, 500],
}
base_model = xgb.XGBClassifier(subsample=0.8, colsample_bytree=0.8,
                               eval_metric='logloss', random_state=42)

grid_search = GridSearchCV(
    estimator=base_model,
    param_grid=param_grid,
    cv=5,
    scoring='roc_auc',
    n_jobs=-1,               # use all CPU cores
    verbose=1
)
t0 = time.perf_counter()
grid_search.fit(X_train, y_train)

print("Best parameters:", grid_search.best_params_)
print(f"Best CV AUC:     {grid_search.best_score_:.4f}")
print(f"Time:            {time.perf_counter() - t0:.0f} s\n")

results = pd.DataFrame(grid_search.cv_results_)
top5 = results.nlargest(5, 'mean_test_score')[[
    'param_max_depth', 'param_learning_rate', 'param_n_estimators', 'mean_test_score']]
print(top5.round(4).to_string(index=False))

Strategy 2 — Random SearchStrategy 2 — Random Search策略 2 — 随机搜索

import time
import xgboost as xgb
from scipy.stats import randint, uniform, loguniform
from sklearn.model_selection import RandomizedSearchCV
from sklearn.metrics import roc_auc_score

# Wide search space: Random Search explores it efficiently
param_dist = {
    'n_estimators':     randint(100, 1000),
    'learning_rate':    loguniform(0.005, 0.3),   # log scale: more small values
    'max_depth':        randint(3, 10),
    'min_child_weight': randint(1, 10),
    'subsample':        uniform(0.5, 0.5),        # uniform(loc, scale) -> [0.5, 1.0]
    'colsample_bytree': uniform(0.4, 0.6),        # -> [0.4, 1.0]
    'gamma':            uniform(0, 2),
    'reg_alpha':        loguniform(1e-4, 10),
    'reg_lambda':       loguniform(1e-4, 10),
}
base = xgb.XGBClassifier(eval_metric='logloss', random_state=42)

random_search = RandomizedSearchCV(
    estimator=base,
    param_distributions=param_dist,
    n_iter=100,              # 100 random combinations
    cv=5,
    scoring='roc_auc',
    n_jobs=-1,
    verbose=1,
    random_state=42
)
t0 = time.perf_counter()
random_search.fit(X_train, y_train)

print("Best parameters:")
for k, v in sorted(random_search.best_params_.items()):
    print(f"  {k:<17}: {v:.4g}" if isinstance(v, float) else f"  {k:<17}: {v}")
print(f"Best CV AUC: {random_search.best_score_:.5f}")
proba = random_search.best_estimator_.predict_proba(X_test)[:, 1]
print(f"Test AUC:    {roc_auc_score(y_test, proba):.5f}")
print(f"Time:        {time.perf_counter() - t0:.0f} s")

Both searches use the same data and the same scoring (ROC-AUC), so the numbers compare directly. A surprise: Grid Search scored slightly higher here (0.9914 vs 0.9906), in a quarter of the time. This synthetic data is easy, and the three parameters in the grid (depth, learning rate, number of trees) are exactly the ones that matter; the grid also happened to include good values. Random Search spent much of its budget on parameters that barely matter for this data. The gap is smaller than the noise between CV folds, so treat it as a tie. Timings are from a 2-core machine; yours will differ.

दोनों searches एक ही data और एक ही scoring (ROC-AUC) इस्तेमाल करती हैं, इसलिए numbers की सीधी तुलना होती है। एक हैरानी: यहाँ Grid Search ने थोड़ा ज़्यादा score किया (0.9914 बनाम 0.9906), वह भी एक-चौथाई समय में। यह synthetic data आसान है, और grid के तीन parameters (depth, learning rate, trees की संख्या) ठीक वही हैं जो मायने रखते हैं; grid में संयोग से अच्छी values भी थीं। Random Search ने अपना बहुत सा budget उन parameters पर खर्च किया जो इस data के लिए मुश्किल से मायने रखते हैं। यह अंतर CV folds के बीच के noise से छोटा है, इसलिए इसे बराबरी मानें। समय 2-core machine का है; आपका अलग होगा।

两种搜索使用相同的数据和评分（ROC-AUC），所以数字可以直接比较。意外的是：这里网格搜索的分数略高（0.9914 vs 0.9906），用时只有四分之一。这份合成数据很简单，网格里的三个参数（深度、学习率、树的数量）恰好就是最重要的那些，而且网格碰巧包含了好的取值。随机搜索把大量预算花在了对这份数据几乎无关紧要的参数上。这个差距比交叉验证各折之间的噪声还小，所以可以视为平局。耗时来自双核机器，你的结果会不同。

Strategy 3 — Optuna (TPE + Pruning)Strategy 3 — Optuna (TPE + Pruning)策略 3 — Optuna（TPE + 剪枝）

Optuna, from Preferred Networks, is a modern tuning framework. TPE splits the trials so far into a "good" group and a "bad" group, models each with a probability density, and samples where good / bad is highest.

Preferred Networks का Optuna एक modern tuning framework है। TPE अब तक के trials को "अच्छे" और "बुरे" group में बाँटता है, हर एक को probability density से model करता है, और वहाँ sample लेता है जहाँ अच्छा / बुरा सबसे ज़्यादा है।

Optuna 是 Preferred Networks 开发的现代调参框架。TPE 把已有的试验分为"好"和"差"两组，分别用概率密度建模，然后在"好/差"比值最高的地方抽样。

pip install optuna

import time
import optuna
import xgboost as xgb
from sklearn.model_selection import cross_val_score
from sklearn.metrics import roc_auc_score

optuna.logging.set_verbosity(optuna.logging.WARNING)   # quiet output

def objective(trial):
    params = {
        'n_estimators':     trial.suggest_int('n_estimators', 100, 1000),
        'learning_rate':    trial.suggest_float('learning_rate', 0.005, 0.3, log=True),
        'max_depth':        trial.suggest_int('max_depth', 3, 10),
        'min_child_weight': trial.suggest_int('min_child_weight', 1, 10),
        'subsample':        trial.suggest_float('subsample', 0.5, 1.0),
        'colsample_bytree': trial.suggest_float('colsample_bytree', 0.4, 1.0),
        'gamma':            trial.suggest_float('gamma', 0, 5),
        'reg_alpha':        trial.suggest_float('reg_alpha', 1e-4, 10, log=True),
        'reg_lambda':       trial.suggest_float('reg_lambda', 1e-4, 10, log=True),
    }
    model = xgb.XGBClassifier(**params, eval_metric='logloss', random_state=42)
    scores = cross_val_score(model, X_train, y_train, cv=5, scoring='roc_auc', n_jobs=-1)
    return scores.mean()                      # Optuna maximises this

study = optuna.create_study(
    direction='maximize',
    sampler=optuna.samplers.TPESampler(seed=42)    # Tree-structured Parzen Estimator
)
t0 = time.perf_counter()
study.optimize(objective, n_trials=100)

# Best score after 10, 30, 60, 100 trials
best_so_far = [max(t.value for t in study.trials[:n]) for n in (10, 30, 60, 100)]
for n, v in zip((10, 30, 60, 100), best_so_far):
    print(f"After {n:>3} trials: best CV AUC = {v:.5f}")

print("\nBest parameters:")
for k, v in study.best_params.items():
    print(f"  {k:<17}: {v:.4g}" if isinstance(v, float) else f"  {k:<17}: {v}")

best_model = xgb.XGBClassifier(**study.best_params, eval_metric='logloss', random_state=42)
best_model.fit(X_train, y_train)
print(f"\nTest AUC: {roc_auc_score(y_test, best_model.predict_proba(X_test)[:, 1]):.5f}")
print(f"Time:     {time.perf_counter() - t0:.0f} s")

Optuna with Pruning (Real Speed-up)Pruning के साथ Optuna (असली Speed-up)带剪枝的 Optuna（真正提速）

pip install optuna-integration

import optuna
import xgboost as xgb
from optuna_integration import XGBoostPruningCallback   # pip install optuna-integration
from sklearn.model_selection import train_test_split

optuna.logging.set_verbosity(optuna.logging.WARNING)
X_tr, X_val, y_tr, y_val = train_test_split(X_train, y_train, test_size=0.15, random_state=42)

def objective_with_pruning(trial):
    model = xgb.XGBClassifier(
        n_estimators=1000,
        learning_rate=trial.suggest_float('learning_rate', 0.005, 0.3, log=True),
        max_depth=trial.suggest_int('max_depth', 3, 10),
        subsample=trial.suggest_float('subsample', 0.5, 1.0),
        colsample_bytree=trial.suggest_float('colsample_bytree', 0.4, 1.0),
        reg_alpha=trial.suggest_float('reg_alpha', 1e-4, 10, log=True),
        eval_metric='logloss',
        early_stopping_rounds=50,
        # The scikit-learn API names the first eval set "validation_0"
        callbacks=[XGBoostPruningCallback(trial, 'validation_0-logloss')],
        random_state=42
    )
    model.fit(X_tr, y_tr, eval_set=[(X_val, y_val)], verbose=False)
    return model.best_score                   # best validation logloss

study2 = optuna.create_study(
    direction='minimize',
    sampler=optuna.samplers.TPESampler(seed=42),
    pruner=optuna.pruners.MedianPruner(n_startup_trials=10, n_warmup_steps=20)
)
study2.optimize(objective_with_pruning, n_trials=80)

pruned = [t for t in study2.trials if t.state == optuna.trial.TrialState.PRUNED]
print(f"Pruned trials: {len(pruned)} of {len(study2.trials)}")
print(f"Best logloss:  {study2.best_value:.5f}")

Strategy 4 — Bayesian Optimisation with a Gaussian ProcessStrategy 4 — Gaussian Process के साथ Bayesian Optimisation策略 4 — 基于高斯过程的贝叶斯优化

Bayesian Optimisation with scikit-optimizescikit-optimize के साथ Bayesian Optimisation用 scikit-optimize 做贝叶斯优化

pip install scikit-optimize

import time
import xgboost as xgb
from skopt import BayesSearchCV                # pip install scikit-optimize
from skopt.space import Real, Integer

search_space = {
    'n_estimators':     Integer(100, 1000),
    'learning_rate':    Real(0.005, 0.3, prior='log-uniform'),
    'max_depth':        Integer(3, 10),
    'min_child_weight': Integer(1, 10),
    'subsample':        Real(0.5, 1.0),
    'colsample_bytree': Real(0.4, 1.0),
    'gamma':            Real(0, 5.0),
    'reg_alpha':        Real(1e-4, 10, prior='log-uniform'),
    'reg_lambda':       Real(1e-4, 10, prior='log-uniform'),
}
base = xgb.XGBClassifier(eval_metric='logloss', random_state=42)

bayes_search = BayesSearchCV(
    estimator=base,
    search_spaces=search_space,
    n_iter=50,               # 50 evaluations
    cv=5,
    scoring='roc_auc',
    n_jobs=-1,
    random_state=42
)

best = []
def on_step(result):                      # result.fun = best (lowest) -AUC so far
    best.append(-result.fun)
    if len(best) in (1, 10, 25, 50):
        print(f"  Iter {len(best):3d} | best CV AUC so far: {-result.fun:.5f}")

t0 = time.perf_counter()
bayes_search.fit(X_train, y_train, callback=[on_step])

print(f"\nBest CV AUC: {bayes_search.best_score_:.5f}")
print("Best parameters:")
for k, v in sorted(bayes_search.best_params_.items()):
    print(f"  {k:<17}: {v:.4g}" if isinstance(v, float) else f"  {k:<17}: {v}")
print(f"Time:        {time.perf_counter() - t0:.0f} s")

Bayesian Optimisation with Optuna (GP Sampler)Optuna के साथ Bayesian Optimisation (GP Sampler)用 Optuna 做贝叶斯优化（GP 采样器）

import optuna
from optuna.samplers import GPSampler   # Gaussian-process sampler (Optuna >= 3.6; needs: pip install torch)

study_gp = optuna.create_study(
    direction='maximize',
    sampler=GPSampler(seed=42),
    study_name='xgb_gp_tuning'
)
study_gp.optimize(objective, n_trials=50)   # same objective as the TPE example
print(f"GP-Optuna best AUC: {study_gp.best_value:.5f}")

This reuses the objective from the TPE example; only the sampler changes. GPSampler needs PyTorch installed (pip install torch).

यह TPE example वाला objective ही इस्तेमाल करता है; सिर्फ़ sampler बदलता है। GPSampler के लिए PyTorch installed होना चाहिए (pip install torch)।

这里复用 TPE 示例中的 objective，只是换了采样器。GPSampler 需要安装 PyTorch（pip install torch）。

Tuning Strategy ComparisonTuning Strategies की तुलना调参策略对比

On the Real Churn Dataअसली Churn Data पर在真实的流失数据上

These numbers come from the code on this page. All four strategies finish within about 0.001 AUC of each other, which is smaller than the noise between CV folds. Grid even comes first, because this synthetic data is easy and its three parameters are the ones that matter. The lesson: a sensible search space plus early stopping matters more than the choice of search method. The method starts to matter when fits are slow and many parameters interact, as in the race simulator above.

ये numbers इस page के code से आए हैं। चारों strategies एक-दूसरे से लगभग 0.001 AUC के अंदर ख़त्म होती हैं, जो CV folds के बीच के noise से भी छोटा है। Grid तो पहले स्थान पर भी है, क्योंकि यह synthetic data आसान है और उसके तीन parameters वही हैं जो मायने रखते हैं। सबक: समझदारी भरा search space और early stopping, search method के चुनाव से ज़्यादा मायने रखते हैं। Method तब मायने रखने लगता है जब fits धीमे हों और कई parameters आपस में असर डालें, जैसा ऊपर race simulator में है।

这些数字来自本页的代码。四种策略的结果彼此相差不到约 0.001 AUC，比交叉验证各折之间的噪声还小。网格搜索甚至排第一，因为这份合成数据很简单，而它的三个参数正是最重要的那些。结论：合理的搜索空间加上 early stopping，比选择哪种搜索方法更重要。当训练很慢、很多参数相互影响时（就像上面的赛跑模拟器），方法的选择才开始变得重要。

Feature Importance in XGBoostXGBoost में Feature ImportanceXGBoost 特征重要性

XGBoost has three built-in importance types. Each measures something different:

XGBoost में तीन built-in importance types हैं। हर एक कुछ अलग मापता है:

XGBoost 有三种内置重要性类型，各自衡量不同的东西：

import pandas as pd
import matplotlib.pyplot as plt
import xgboost as xgb
import shap                                     # pip install shap

# `model`, `X_test` come from the churn example above
feature_names = [f"f{i}" for i in range(X_test.shape[1])]
model.get_booster().feature_names = feature_names

gain = model.get_booster().get_score(importance_type='gain')
imp_df = (pd.DataFrame({'feature': list(gain), 'gain': list(gain.values())})
            .sort_values('gain', ascending=False))
print(imp_df.head(5).round(2).to_string(index=False))

xgb.plot_importance(model, importance_type='gain', max_num_features=15)
plt.title('XGBoost Feature Importance (Gain)')
plt.tight_layout()
plt.savefig('importance.png')

# SHAP values: the most reliable view of each feature's effect
explainer = shap.TreeExplainer(model)
shap_values = explainer.shap_values(X_test)
shap.summary_plot(shap_values, X_test, feature_names=feature_names, show=False)
mean_abs = abs(shap_values).mean(axis=0)
top = sorted(zip(feature_names, mean_abs), key=lambda kv: -kv[1])[:5]
print("\nTop 5 by mean |SHAP|: " + ", ".join(f"{f} ({v:.3f})" for f, v in top))

Complete Production Pipeline — End-to-Endपूरी Production Pipeline — शुरू से अंत तक完整生产流程 — 端到端

This pipeline fixes a common mistake. Many tutorials find the number of trees once (at learning_rate 0.05), then let Optuna change the learning rate while keeping that tree count fixed. But the right number of trees depends on the learning rate. Here, every trial uses early stopping on the validation set, so it finds its own tree count.

यह pipeline एक आम गलती ठीक करती है। कई tutorials trees की संख्या एक बार खोजते हैं (learning_rate 0.05 पर), फिर Optuna को learning rate बदलने देते हैं जबकि trees की संख्या वही रहती है। पर trees की सही संख्या learning rate पर निर्भर करती है। यहाँ हर trial validation set पर early stopping इस्तेमाल करता है, इसलिए वह अपनी trees की संख्या खुद खोजता है।

这个流程修正了一个常见错误。很多教程只找一次树的数量（在 learning_rate 0.05 时），然后让 Optuna 改变学习率，却保持树的数量不变。但合适的树数量取决于学习率。这里每个试验都在验证集上使用 early stopping，因此会找到自己的树数量。

import optuna
import xgboost as xgb
from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split
from sklearn.metrics import roc_auc_score, classification_report

# ── STEP 1: Data: train / validation / test ──────────────
X, y = make_classification(n_samples=10000, n_features=25,
                           n_informative=15, random_state=42)
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, stratify=y, random_state=42)
X_tr, X_val, y_tr, y_val = train_test_split(
    X_train, y_train, test_size=0.15, stratify=y_train, random_state=42)

# ── STEP 2: Optuna. Each trial uses early stopping on the validation set,
#    so the number of trees is found per trial (it depends on learning_rate).
optuna.logging.set_verbosity(optuna.logging.WARNING)

def objective(trial):
    model = xgb.XGBClassifier(
        n_estimators=3000,
        learning_rate=trial.suggest_float('learning_rate', 0.01, 0.2, log=True),
        max_depth=trial.suggest_int('max_depth', 3, 8),
        min_child_weight=trial.suggest_int('min_child_weight', 1, 7),
        subsample=trial.suggest_float('subsample', 0.6, 1.0),
        colsample_bytree=trial.suggest_float('colsample_bytree', 0.5, 1.0),
        gamma=trial.suggest_float('gamma', 0, 3),
        reg_alpha=trial.suggest_float('reg_alpha', 1e-4, 5, log=True),
        reg_lambda=trial.suggest_float('reg_lambda', 1e-4, 5, log=True),
        eval_metric='logloss', early_stopping_rounds=50, random_state=42)
    model.fit(X_tr, y_tr, eval_set=[(X_val, y_val)], verbose=False)
    trial.set_user_attr('n_trees', model.best_iteration + 1)
    return model.best_score                       # validation logloss

study = optuna.create_study(direction='minimize',
                            sampler=optuna.samplers.TPESampler(seed=42))
study.optimize(objective, n_trials=80)
best_n = study.best_trial.user_attrs['n_trees']
print(f"Best val logloss: {study.best_value:.5f} with {best_n} trees")

# ── STEP 3: Refit on train + validation with the best settings ──
final_model = xgb.XGBClassifier(**study.best_params, n_estimators=best_n,
                                eval_metric='logloss', random_state=42)
final_model.fit(X_train, y_train)

# ── STEP 4: Test set: used exactly once ──────────────────
y_proba = final_model.predict_proba(X_test)[:, 1]
print(f"\nTest AUC-ROC: {roc_auc_score(y_test, y_proba):.5f}\n")
print(classification_report(y_test, final_model.predict(X_test), digits=4))

Golden Rules — XGBoost & Hyperparameter TuningGolden Rules — XGBoost और Hyperparameter Tuning黄金法则 — XGBoost 与超参数调优

XGBoost vs LightGBM vs CatBoost — Quick Decision GuideXGBoost vs LightGBM vs CatBoost — जल्दी फ़ैसलाXGBoost vs LightGBM vs CatBoost — 快速选择指南