Gradient Boosting
Press Next → or use ← → arrow keys
The Idea — Learn Only From Mistakes
Gradient Boosting is that study method as an algorithm. Each new tree ignores what the ensemble already predicts well and trains only on the residual errors left behind. Nobody re-teaches the whole problem — each round just corrects what's still wrong.
Gradient Boosting is a sequential ensemble of shallow trees where each tree is trained to predict the residual mistakes of all the trees before it — so the ensemble steadily reduces error, correcting bias one small step at a time.
Why "Gradient"? · Bagging vs Boosting
The residual a tree fits is really the negative gradient of the loss with respect to the current prediction — the steepest downhill direction. So each tree takes one gradient-descent step, but in function space. Swap the loss and the same machinery handles regression, classification, or ranking.
| Property | Bagging (Random Forest) | Boosting (Gradient Boosting) |
|---|---|---|
| Tree order | Parallel · independent | Sequential · dependent |
| Each tree trains on | Random bootstrap sample | Residuals of prior trees |
| Tree depth | Deep (low bias, high var) | Shallow (depth 3–5) |
| Reduces | Variance only | Bias and variance |
| Parallelizable | Yes, trivially | No, inherently sequential |
The Gradient Boosting Loop
F₀(x).rᵢ = yᵢ − F(xᵢ) — how wrong the model is right now.Fₘ = Fₘ₋₁ + η·hₘ — scale by learning rate η so no tree dominates.F(x) = F₀ + Σ η·hₘ(x) — just a scaled sum, no voting.Unlike AdaBoost's weighted vote, gradient boosting simply adds each scaled tree to the running prediction. The "correction" lives entirely in what each tree is trained on — the residuals.
The Sequential Process, Visualized
The residual flows down to train the next shallow tree; the scaled tree flows back up to update the ensemble. Early trees set the foundation and every later tree refines it — the error-correction chain that gives boosting its power.
Watch One Prediction Climb To The Truth
Start at the mean, £280k (residual +£120k). Tree 1 predicts the residual; add η×it →
£291k. Tree 2 → £301k. With η = 0.1, each round nudges
the prediction a fraction of the remaining error, so after ~100 rounds it lands near £400k.
Small, precise steps beat one reckless leap.
Residual Error Collapses Each Round
For a single house, the residual falls from £120k → £18k in six rounds. Every tree removes a slice of what's left, so the error decays toward zero. This is the whole method in one picture: relentless, incremental correction.
Gradient Boosting In Four Equations
For squared-error loss the negative gradient is the raw residual y − F. But for
log-loss, absolute error, or Huber, the gradient is a different expression. "Pseudo-residual" is the
generic name that covers every loss — it's always the negative gradient the next tree chases.
Swap The Loss, Change The Task
| Loss | Task | Pseudo-Residual | Character |
|---|---|---|---|
| Squared error | Regression (default) | yᵢ − Fᵢ | Sensitive to outliers |
| Absolute error | Regression | sign(yᵢ − Fᵢ) | Robust · targets median |
| Huber | Regression | MSE small · MAE large | Best of both worlds |
| Log-loss | Classification | yᵢ − p̂ᵢ | Default for probabilities |
| Exponential | Classification | reduces to AdaBoost | Very outlier-sensitive |
| Quantile | Regression intervals | asymmetric | Prediction ranges |
Because the tree only ever fits the negative gradient, changing the loss is all it takes to go from house-price regression (squared error) to fraud detection (log-loss) to robust modelling on messy data (Huber) — or even confidence intervals (quantile). Match the loss to your evaluation metric.
Learning Rate × Trees — The Golden Trade-off
Low η Converges Smoothly · High η Overshoots
| Lowering the learning rate… | Effect |
|---|---|
| Trees needed | More (higher n_estimators) |
| Compute cost | Increases |
| Generalization | Better |
| Overfitting | Reduced |
Add Randomness — Stochastic Gradient Boosting
Introduced by Friedman, row subsampling is almost always a net win: it speeds up each round, adds
diversity, and improves validation accuracy on real, noisy datasets. Pair it with
max_features for feature subsampling too.
Six Ways To Stop Overfitting, Ranked
Unlike Random Forest — where more trees never hurt — adding trees to gradient boosting will eventually overfit if η isn't low enough. Beyond the optimal point, new trees fit noise in the residuals, not signal. That's why early stopping is non-negotiable.
Regression vs Classification
squared_error, huber, quantile. Pseudo-residual = raw residual yᵢ − Fᵢ. Tasks: prices, energy demand.log_loss. Pseudo-residual = yᵢ − p̂ᵢ. Outputs probabilities via predict_proba(). Tasks: fraud, diagnosis.Whether you're predicting a price or a probability, the loop is identical: initialize, compute the gradient, fit a tree to it, add with η, repeat. Only the loss function — and therefore the shape of the pseudo-residual — differs.
sklearn In Practice — Annotated
from sklearn.ensemble import GradientBoostingClassifier gb = GradientBoostingClassifier( n_estimators=500, # set high — early stopping prunes it learning_rate=0.05, # small steps, better generalisation max_depth=4, # shallow trees prevent overfitting min_samples_leaf=10, # regularise leaf size subsample=0.75, # stochastic GB: 75% rows per tree max_features=0.5, # 50% of features per split loss='log_loss', # classification objective validation_fraction=0.1, # hold-out for early stopping n_iter_no_change=20, # stop after 20 flat rounds random_state=42) gb.fit(X_train, y_train) print(gb.n_estimators_) # e.g. 312 — stopped itself · Test AUC ≈ 0.974
Fix η → tune max_depth + min_samples_leaf → tune subsample +
max_features → then lower η further and add trees. For regression, swap in
GradientBoostingRegressor with loss='huber' for outlier-robust fits.
Where Gradient Boosting Aims
Random Forest starts accurate but scattered and averages the scatter away (kills variance). Gradient Boosting starts simple but off-centre and iteratively corrects toward the target (kills bias) while shrinkage and shallow trees hold variance down — landing tight and on the bull.
Gradient Boosting vs Random Forest
| Property | Gradient Boosting | Random Forest |
|---|---|---|
| Core idea | Sequential error correction | Parallel variance averaging |
| Reduces | Bias + variance | Variance only |
| Tree depth | Shallow (3–5) · weak | Deep · complex |
| Parallelizable | No — sequential | Yes — trivially |
| Overfitting risk | High — tune carefully | Low — self-regularizing |
| Tuning sensitivity | High — many knobs | Low — good defaults |
| Peak tabular accuracy | Highest in practice | Very good, rarely highest |
| Best use case | Max accuracy · competitions | Fast, robust baseline |
Start with Random Forest. If it misses your accuracy target after basic tuning, move to Gradient Boosting. And if your dataset tops ~100,000 rows, skip straight to LightGBM — sklearn's GB is too slow at that scale.
LightGBM — Same Logic, Built For Scale
| Property | Standard GB | LightGBM |
|---|---|---|
| Tree growth | Level-wise (balanced) | Leaf-wise (max-gain first) |
| Split finding | Exact — every value | Histogram — 256 bins |
| Speed on 1M rows | Slow | 10–100× faster |
| Categoricals | Manual encoding | Native support |
Reading Feature Importance
Built-in gain and split-count are fine for a quick look, but they can mislead with correlated features. For stakeholder reports, audits, or production monitoring, SHAP (TreeExplainer) is the trustworthy, direction-aware choice.
Eight Non-Negotiable Rules
You Now Own Gradient Boosting
Gradient boosting turns a chain of weak, shallow trees into a champion by having each one descend the loss gradient — fitting the residual mistakes of the trees before it. Shrinkage keeps the steps small, shallow trees and subsampling keep variance down, and early stopping calls it at the right moment.
Compare XGBoost (regularized, second-order), LightGBM (leaf-wise, fast) and CatBoost (native categoricals). Tune with Optuna, explain with SHAP, and practise on Kaggle's tabular competitions — this family rules them.
📉 End of tutorial · Press ← to review, or click Restart