Machine Learning 📂 Supervised Learning · 10 of 19 172 min read

Random Forest

A comprehensive, story-driven tutorial on Random Forests — covering the core intuition, bias-variance theory, feature importance, OOB validation, fraud detection case study, and production-ready Python code.

🌐 Language
▶️
This Tutorial Has Seven Simulators

Look for the interactive boxes: Ask the Audience (Section 01), One Tree vs a Forest (03), Bootstrap Sampler (04), More Trees & the OOB Score (08), MDI vs Permutation Importance (09), max_features & Tree Correlation (10) and Imbalanced Classes (12). Every forest in them is trained live in your browser.

Section 01

The Story That Explains Random Forest

📖 Real-World Analogy
Who Wants to Be a Millionaire — Ask the Audience
You are on a quiz show. The question is hard. You have two lifelines left: Phone a Friend (one very smart person) or Ask the Audience (500 ordinary people vote). Which one do you trust more?

The audience usually wins. No single person in it is smarter than your friend. But each person makes different mistakes, and when 500 people vote, those mistakes go in all directions and cancel out. Your friend has only one brain — one blind spot can cost you everything.

That is the whole idea behind Random Forest.

A Random Forest is a team of many decision trees. Each tree learns from a slightly different random sample of the data, and each split looks at only a random few features. To predict, every tree gives an answer. For classification the majority vote wins. For regression the answers are averaged. No tree has to be perfect — the team is strong because the trees make different mistakes.

🌲
The Core Idea

Train many models that are decent and different, then combine their answers. The combined answer is usually more accurate and more stable than any single model. This is called ensemble learning, and Random Forest is one of its most reliable forms. There is one condition: the models must not all make the same mistakes.

Try It — Ask the Audience

Each voter is right only 60% of the time. Add more voters and watch the majority become almost always right. Then raise ρ (how alike the voters think) and see the magic disappear.

🙋 Ask the AudienceEach voter is right with probability p. The crowd answers by majority vote.
number of voters (trees)
each voter is right (p)
how alike the voters think (ρ)
● right vote   ● wrong vote   (up to 101 voters shown)
Section 02

The Foundation — Decision Trees (Quick Recap)

Before building a forest, we need one tree. A decision tree splits the data by asking yes/no questions, one feature at a time, until it reaches a prediction.

🌳 A Single Decision Tree — Predicting Loan Default
Root
Income < £30,000? → YES goes left, NO goes right
Node
Credit Score < 600? → splits each branch again
Node
Loan Amount > £50,000? → splits further
Leaf
Final prediction: Default or No Default
⚠️
The Big Weakness of One Deep Tree

A deep tree keeps splitting until every leaf is pure. So it memorises the training data — the noise, the outliers, even one strange row. Training accuracy is close to 100%, but accuracy on new data is much lower. This is overfitting. Worse, a small change in the data can give a completely different tree. We say the tree has high variance.

Section 03

The Two Ingredients of Random Forest

Random Forest fixes the variance problem by adding randomness in two separate ways. Both make the trees different from each other.

🎲
Ingredient 1 — Bagging
Bootstrap Aggregating
Each tree trains on a bootstrap sample: rows drawn at random with replacement. With 1,000 rows, each tree sees about 632 unique rows; the rest of its sample is repeats. Different data → different trees → different mistakes.
🎯
Ingredient 2 — Feature Randomness
Random Subspace Method
At each split, the tree may only look at a random subset of features (max_features). scikit-learn default: √p for classification and all p (1.0) for regression. Many people use p/3 for regression, but you must set it yourself.
🗳️
The Aggregation
Majority Vote / Average
After training 100–500 trees, every tree predicts. Classification: majority vote (scikit-learn actually averages the trees’ class probabilities). Regression: the mean of all predictions. Individual mistakes cancel out.
🔑
Why Both Ingredients Help

With bagging alone, every tree can still use every feature. One strong feature then wins the first split in almost every tree, so the trees look alike. With feature randomness alone, every tree sees the same rows. Together they give truly different trees, whose mistakes cancel instead of adding up.

Try It — One Tree vs a Forest

Start with 1 tree and look at its jagged border. Then use 200 trees. Click on the map to see how the trees vote for one point. Untick both boxes to see what happens with no randomness.

🌳 One Tree vs a ForestTwo-moons data, 160 training points. Colour = what the model predicts.
number of trees used
● class A   ■ class B   darker colour = more trees agree   ★ your point
Section 04

How Bootstrap Sampling Works

📖 Story
The Hat Trick
A bag holds tickets numbered 1 to 10. For each tree, you pull a ticket, write down its number, and put the ticket back. After 10 pulls you have 10 numbers — some repeated, some missing. This is sampling with replacement. Every tree gets its own sample, so every tree sees a different “view” of the data.
Tree Bootstrap Sample (row numbers) Out-of-Bag Rows (never picked)
Tree 11, 3, 3, 5, 7, 7, 9, 2, 4, 46, 8, 10
Tree 22, 4, 6, 6, 8, 1, 3, 9, 9, 57, 10
Tree 310, 2, 5, 5, 1, 8, 3, 7, 6, 64, 9
Tree 47, 9, 1, 4, 4, 6, 2, 10, 3, 85
🎟️ The 63.2% Rule
P(row is picked at least once) = 1 − (1 − 1/N)ᴺ → 1 − 1/e ≈ 0.632
Each draw misses a given row with probability (1 − 1/N). After N draws it is still missing with probability (1 − 1/N)ᴺ ≈ 36.8%. So each tree sees about 63.2% of the unique rows.

Try It — Bootstrap Sampler

Draw the tickets one at a time, or a full sample at once. Orange tickets were never picked — they are the out-of-bag rows. Then repeat 1000 times to see the 63.2% rule appear.

🎟️ Bootstrap SamplerDraw N tickets from a bag of N, putting each ticket back after drawing it.
dataset size N
■ picked (×2 = picked twice)   ▢ never picked = out-of-bag (OOB)
🎁
A Free Test Set — the Out-of-Bag (OOB) Score

Every row is out-of-bag for about 37% of the trees. To score a row, use only the trees that never saw it and let them vote. Do this for every row and compare with the true labels — that is the OOB score. It is a nearly unbiased estimate of accuracy on new data, and you get it without a separate validation set. Set oob_score=True in scikit-learn.

Section 05

Visual Diagram — Inside a Random Forest

01
Full Training Dataset
Start with the labelled data — for example 10,000 rows × 20 features. All bootstrap samples are drawn from this pool.
02
Bootstrap Sampling × N Trees
For each of the N trees (say 300), draw 10,000 rows with replacement. Each sample is different, and each one trains exactly one tree.
03
Random Features at Each Split
At every node, the tree tries only a random few features — with 'sqrt', int(√20) = 4 of the 20 — and picks the best split among them.
04
Grow Each Tree Deep
Trees are usually grown deep with no pruning. Each deep tree has low bias but high variance — the averaging step removes much of that variance.
05
Combine the Predictions (Vote / Average)
For a new row, all 300 trees predict. Classification → the class with the most support. Regression → the mean of the 300 numbers.
Section 06

Classification vs Regression

🗳️ Classification (Voting)
TreePrediction
Tree 1Spam
Tree 2Spam
Tree 3Not Spam
Tree 4Spam
Tree 5Spam
FinalSpam (4 of 5 votes)
📊 Regression (Averaging)
TreePredicted Price
Tree 1£312,000
Tree 2£298,500
Tree 3£325,000
Tree 4£310,000
Tree 5£304,500
Final£310,000 (mean)
💡
Votes Become Probabilities

In the spam example, 4 of 5 trees say “Spam”, so the forest is about 80% sure. predict_proba() gives you this number. You can use it to rank customers or to move the decision threshold (see Section 12).

Section 07

Python Implementation

Classification — Predicting Customer Churn

from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import train_test_split
from sklearn.metrics import classification_report
from sklearn.datasets import make_classification

# Simulated churn data: 5,000 customers, 15 features
X, y = make_classification(n_samples=5000, n_features=15, n_informative=10,
                           n_redundant=3, random_state=42)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

rf = RandomForestClassifier(
    n_estimators=300,        # number of trees
    max_features='sqrt',     # features tried per split -> sqrt(15) ~ 3
    max_depth=None,          # grow trees fully
    min_samples_leaf=1,      # smallest allowed leaf
    bootstrap=True,          # bagging on
    oob_score=True,          # free out-of-bag estimate
    n_jobs=-1,               # use all CPU cores
    random_state=42
)
rf.fit(X_train, y_train)

print(f"OOB score (free estimate): {rf.oob_score_:.4f}")
print(f"Test accuracy:             {rf.score(X_test, y_test):.4f}\n")
print(classification_report(y_test, rf.predict(X_test), digits=3))
Output
OOB score (free estimate): 0.9545 Test accuracy: 0.9480 precision recall f1-score support 0 0.939 0.958 0.949 500 1 0.957 0.938 0.947 500 accuracy 0.948 1000 macro avg 0.948 0.948 0.948 1000 weighted avg 0.948 0.948 0.948 1000

The OOB score (95.5%) and the test accuracy (94.8%) are very close. The OOB score was free — it used no test data.

Regression — Predicting House Prices

We use the California housing data (one row per district, 1990 census). Note that total_bedrooms has 207 missing values.

import pandas as pd
from sklearn.ensemble import RandomForestRegressor
from sklearn.model_selection import train_test_split
from sklearn.metrics import mean_absolute_error, r2_score

# California housing (1990 census), 20,640 districts
url = "https://raw.githubusercontent.com/ageron/handson-ml2/master/datasets/housing/housing.csv"
df = pd.read_csv(url)
X = df.drop(columns=["median_house_value", "ocean_proximity"])
y = df["median_house_value"]
print(f"Missing values in total_bedrooms: {X['total_bedrooms'].isna().sum()}")

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

rf_reg = RandomForestRegressor(
    n_estimators=300,
    max_features=1.0,        # regression default: try all features
    oob_score=True,
    n_jobs=-1,
    random_state=42
)
rf_reg.fit(X_train, y_train)     # scikit-learn >= 1.4 accepts the NaN values
y_pred = rf_reg.predict(X_test)

print(f"OOB R²:  {rf_reg.oob_score_:.4f}")
print(f"Test R²: {r2_score(y_test, y_pred):.4f}")
print(f"MAE:     ${mean_absolute_error(y_test, y_pred):,.0f}")
Output
Missing values in total_bedrooms: 207 OOB R²: 0.8185 Test R²: 0.8125 MAE: $31,963
⚡
Two Practical Tips

Speed: set n_jobs=-1. The trees are independent, so they can be built on all CPU cores at the same time. On an 8-core machine this is often 5–8× faster, with exactly the same results.
Missing values: since scikit-learn 1.4, Random Forest accepts NaN directly — that is why the code above works without imputation. In older versions you must fill the gaps first.

Section 08

Key Hyperparameters

ParameterDefaultWhat It ControlsTuning Advice
n_estimators100Number of treesMore is better until the error goes flat. 300–500 is a good start.
max_features'sqrt' (classifier)
1.0 (regressor)		Features tried at each splitUsually the most important one to tune.
max_depthNoneMaximum depth of each treeNone = grow fully. Limit it to save memory or to smooth noisy data.
min_samples_leaf1Smallest number of rows in a leafTry 2–10 on noisy data. Also gives smoother probabilities.
min_samples_split2Rows needed to split a nodeRarely needed if you tune min_samples_leaf.
bootstrapTrueUse bagging or notKeep True. False removes one source of diversity and the OOB score.
max_samplesNoneSize of each bootstrap samplee.g. 0.5 = each tree sees half the rows. Faster on big data.
oob_scoreFalseCompute the OOB scoreSet True — a free accuracy check.
class_weightNoneWeights for rare classes'balanced' or 'balanced_subsample' for imbalanced data.

Try It — More Trees & the OOB Score

The forest grows from 1 to 300 trees. Watch the test error and the OOB error fall and then go flat. They never turn upward.

📈 More Trees & the OOB ScoreTwo-moons data, 300 training rows. Error as the forest grows from 1 to 300 trees.
look at this many trees

Hyperparameter Tuning with RandomizedSearchCV

Random search tries a fixed number of random combinations. It is much faster than trying every combination (grid search) and usually finds an equally good answer.

# Tune the churn classifier: X_train, y_train come from the classification example
from scipy.stats import randint
from sklearn.model_selection import RandomizedSearchCV

param_dist = {
    "n_estimators":      randint(100, 400),
    "max_features":      ["sqrt", "log2", 0.3, 0.5],
    "max_depth":         [None, 10, 20],
    "min_samples_leaf":  randint(1, 6),
}
search = RandomizedSearchCV(
    RandomForestClassifier(n_jobs=-1, random_state=42),
    param_distributions=param_dist,
    n_iter=20,            # try 20 random combinations
    cv=5,
    scoring="f1",
    n_jobs=-1,
    random_state=42,
)
search.fit(X_train, y_train)
print("Best params:", search.best_params_)
print(f"Best CV F1:   {search.best_score_:.4f}")
print(f"Test F1:      {search.score(X_test, y_test):.4f}")
Output
Best params: {'max_depth': 20, 'max_features': 0.3, 'min_samples_leaf': 1, 'n_estimators': 149} Best CV F1: 0.9549 Test F1: 0.9519

The tuned model reaches a test F1 of 0.952 — a small gain over the defaults. This is typical: Random Forest works well out of the box, and tuning usually adds only a little.

Section 09

Feature Importance

A big practical advantage of Random Forest is that it can rank the features. The built-in method is Mean Decrease in Impurity (MDI): every time a feature is used for a split, we add up how much it reduced the Gini impurity (or the MSE for regression). Then we average over all trees. This is what feature_importances_ gives you. 

import numpy as np
import pandas as pd

feature_names = X.columns

# ── MDI importance (built in, computed on the training data) ──
mdi = pd.DataFrame({
    "feature":    feature_names,
    "importance": rf_reg.feature_importances_,
    "std":        np.std([t.feature_importances_ for t in rf_reg.estimators_], axis=0),
}).sort_values("importance", ascending=False)
print(mdi.round(4).to_string(index=False))
Output
feature importance std median_income 0.5208 0.0083 longitude 0.1561 0.0114 latitude 0.1475 0.0115 housing_median_age 0.0654 0.0056 population 0.0377 0.0066 total_bedrooms 0.0284 0.0061 total_rooms 0.0243 0.0029 households 0.0197 0.0034
⚠️
MDI Has Two Known Problems

1. It is computed on the training data, so it can reward features that only helped the trees memorise noise.
2. It is biased toward high-cardinality features — numbers with many unique values, like an ID column — because they offer many possible split points.
Permutation importance is less biased: shuffle one column in the test data and measure how much the score drops. It costs more compute.

Permutation Importance

from sklearn.inspection import permutation_importance

result = permutation_importance(rf_reg, X_test, y_test,
                                n_repeats=10, n_jobs=-1, random_state=42)
perm = pd.DataFrame({
    "feature":    feature_names,
    "importance": result.importances_mean,     # drop in R² when the column is shuffled
    "std":        result.importances_std,
}).sort_values("importance", ascending=False)
print(perm.round(4).to_string(index=False))
Output
feature importance std latitude 0.8934 0.0166 longitude 0.8028 0.0118 median_income 0.6849 0.0177 housing_median_age 0.0668 0.0026 population 0.0506 0.0036 total_bedrooms 0.0259 0.0029 total_rooms 0.0193 0.0010 households 0.0050 0.0006

The two methods disagree! MDI says median_income is by far the most important. Permutation importance puts latitude and longitude first. Why? Latitude and longitude work as a team — together they say where the house is. Shuffle one of them, and the location becomes nonsense, so R² falls a lot. MDI splits the credit between many small splits. Neither number is “the truth”; they answer different questions. When features are correlated, read both with care.

Try It — MDI vs Permutation Importance

The target depends only on income and credit. Add a useless customer_id column (a unique random number per row) and see which method is fooled.

🔍 MDI vs Permutation ImportanceThe target depends only on income (strong) and credit (weaker). Add useless columns and compare.
MDI (feature_importances_) — from training
Permutation — accuracy drop on test data
Section 10

Bias–Variance: Single Tree vs Forest

📐
The Ensemble Maths

Suppose we have N trees. Each has variance σ², and any two trees have correlation ρ. The variance of their average is:

Var(forest) = ρ·σ² + (1−ρ)/N · σ²

As N grows, the second term goes to zero. The first term ρ·σ² stays — no number of trees can remove it. So making the trees less correlated (smaller ρ) matters as much as adding trees. That is exactly what max_features does. Averaging does not change the bias: the forest’s bias is about the same as one tree’s.

Try It — max_features & Tree Correlation

Move max_features from 1 to 8. Watch three things: how good one tree is, how alike the trees are (ρ), and how good the forest is. Press Compute all 8 to draw the full curves.

🎲 max_features & Tree Correlation8 features. Feature 1 is the strongest. 40 trees per forest.
max_features (features tried at each split)
ModelBiasVarianceOverfit RiskTypical Use
Single shallow treeHighLowLowSimple, readable rules
Single deep treeLowVery HighSevereRarely useful alone
Random ForestLow (same as a deep tree)LowLowGeneral-purpose workhorse ✅
Gradient BoostingVery LowMediumModerateTop accuracy, needs tuning
Section 11

When to Use Random Forest

✅
Tabular Data
Random Forest is excellent on table data with numeric and categorical columns. It is often the first model people try.
surveys, logs, transactions, sensors
✅
Little Preprocessing
No scaling or normalisation needed. Categories must be encoded as numbers, but simple ordinal codes usually work fine for trees.
no StandardScaler needed
✅
A Quick Feature Ranking
Use feature importance (preferably permutation importance) to find and drop useless columns before trying a more expensive model.
feature screening
❌
Very High-Dimensional Sparse Data
Text as bag-of-words with 50,000+ columns is hard for Random Forest. Linear models or gradient boosting with sparse support work better.
text, one-hot with many categories
❌
Extreme Real-Time Speed or Small Memory
500 deep trees need a lot of memory and more time per prediction than one small model. For tiny devices, use a single tree or a linear model.
edge devices, huge streams
❌
Full Explanation Required
You cannot read 300 trees like a rule book. If every decision must be explained in plain words, use a shallow tree or logistic regression.
regulated credit, medical rules
❌
Predicting Beyond the Data
A regression forest can only output averages of training targets. It cannot predict a value higher than the highest one it has seen — so it cannot follow a trend into the future.
extrapolation, time trends
Section 12

Handling Class Imbalance

Random Forest can struggle when one class is rare (fraud detection: 97% normal, 3% fraud). Most bootstrap samples contain very few fraud rows, so most trees learn to say “normal”. Here are the options:

⚖️ Imbalance Fixes for Random Forest
Option 1
class_weight='balanced' — rare classes get bigger weights (inverse to their frequency). Easy; no change to the data.
Option 2
class_weight='balanced_subsample' — the same, but the weights are recomputed for each tree’s bootstrap sample.
Option 3
BalancedRandomForestClassifier from the imbalanced-learn package (pip install imbalanced-learn) — each tree trains on a balanced sample with the majority class undersampled.
Option 4
Move the decision threshold. Use predict_proba() and predict “fraud” when the score is above 0.3 instead of 0.5. This raises recall on the rare class. 
from sklearn.datasets import make_classification
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import train_test_split
from sklearn.metrics import precision_score, recall_score
from imblearn.ensemble import BalancedRandomForestClassifier

# Fraud-like data: about 3% positives
X, y = make_classification(n_samples=20000, n_features=20, n_informative=6,
                           weights=[0.97, 0.03], class_sep=0.8, random_state=7)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25,
                                                    stratify=y, random_state=7)

def report(name, y_pred):
    print(f"{name:28s} recall={recall_score(y_test, y_pred):.3f}  "
          f"precision={precision_score(y_test, y_pred):.3f}")

rf = RandomForestClassifier(n_estimators=300, n_jobs=-1, random_state=42).fit(X_train, y_train)
report("Plain RF", rf.predict(X_test))

rf_bal = RandomForestClassifier(n_estimators=300, class_weight="balanced_subsample",
                                min_samples_leaf=5, n_jobs=-1, random_state=42).fit(X_train, y_train)
report("class_weight + leaf=5", rf_bal.predict(X_test))

brf = BalancedRandomForestClassifier(n_estimators=300, sampling_strategy="all",
                                     replacement=True, bootstrap=False,
                                     n_jobs=-1, random_state=42).fit(X_train, y_train)
report("BalancedRandomForest", brf.predict(X_test))

proba = rf.predict_proba(X_test)[:, 1]
report("Plain RF, threshold 0.3", (proba >= 0.3).astype(int))
Output
Plain RF recall=0.159 precision=1.000 class_weight + leaf=5 recall=0.276 precision=0.887 BalancedRandomForest recall=0.676 precision=0.308 Plain RF, threshold 0.3 recall=0.382 precision=0.833

Plain RF catches only 16% of fraud. Class weights help a little (28%). The balanced forest catches 68%, but now most of its alarms are false (precision 0.31). Simply lowering the threshold to 0.3 gives a good middle ground. There is no free lunch: you always trade precision for recall. Choose the threshold from the real cost of each mistake.

Try It — Imbalanced Classes

Fraud is only about 4% of the rows. Slide the threshold and watch caught fraud and false alarms change. Then switch to a balanced bootstrap.

⚖️ Imbalanced ClassesFraud is only about 4% of rows. Move the threshold and compare plain and balanced forests.
decision threshold
Section 13

A Complete Real-World Example — Titanic Survival

import pandas as pd
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import cross_val_score

url = "https://raw.githubusercontent.com/datasciencedojo/datasets/master/titanic.csv"
df = pd.read_csv(url)                       # 891 passengers

# Simple preprocessing (no scaling needed for trees)
df["Age"] = df["Age"].fillna(df["Age"].median())
df["Embarked"] = df["Embarked"].fillna("S")
df["Sex"] = df["Sex"].map({"male": 0, "female": 1})
df["Embarked"] = df["Embarked"].map({"S": 0, "C": 1, "Q": 2})

features = ["Pclass", "Sex", "Age", "SibSp", "Parch", "Fare", "Embarked"]
X, y = df[features], df["Survived"]

rf = RandomForestClassifier(n_estimators=500, max_features="sqrt", min_samples_leaf=2,
                            oob_score=True, n_jobs=-1, random_state=42)

cv_scores = cross_val_score(rf, X, y, cv=5, scoring="accuracy")
print(f"CV accuracy: {cv_scores.mean():.4f} +/- {cv_scores.std():.4f}")

rf.fit(X, y)
print(f"OOB score:   {rf.oob_score_:.4f}\n")
for feat, imp in sorted(zip(features, rf.feature_importances_), key=lambda x: -x[1]):
    print(f"  {feat:9s}: {imp:.4f}")
Output
CV accuracy: 0.8317 +/- 0.0307 OOB score: 0.8316 Sex : 0.3318 Fare : 0.2435 Age : 0.1997 Pclass : 0.1096 SibSp : 0.0469 Parch : 0.0357 Embarked : 0.0328

Cross-validation (83.2%) and the OOB score (83.2%) agree almost perfectly. Sex is the strongest feature (“women and children first”). Remember the MDI warning: Fare and Age have many unique values, so MDI may rank them a bit too high.

🎯
No Scaling Needed

We did not scale Age, Fare or any other feature. Trees split on thresholds (“Age < 9.5?”), not on distances, so the scale of a feature does not matter — only the order of its values. This makes Random Forest easier to use than SVMs, KNN or neural networks, which all need careful scaling.

Section 14

Random Forest vs Gradient Boosting — Quick Comparison

PropertyRandom ForestGradient Boosting (XGBoost / LightGBM)
How trees are builtIn parallel — independentOne after another — each fixes the previous errors
Tree typeDeep trees (low bias); averaging cuts varianceShallow trees (high bias); boosting cuts bias
Overfitting riskLow — bagging protects wellHigher — needs learning rate and early stopping
Training speedFast (parallel)Sequential, but modern libraries are very fast
Hyperparameter sensitivityLow — defaults work wellHigh — needs tuning to shine
Peak accuracyVery goodOften the best on tabular data
Missing valuesSupported in scikit-learn ≥ 1.4Handled natively by XGBoost, LightGBM, HistGradientBoosting
Best choice when…You need a fast, robust baselineYou need the last bit of accuracy
🏆
The Practitioner’s Rule

Start with Random Forest as your baseline — it is fast, robust and good with almost no tuning. If you need the last 1–3% of accuracy, try LightGBM or XGBoost and tune carefully. Many production systems keep Random Forest because the small gain from boosting is not worth the extra work.

Section 15

Golden Rules

🌲 Random Forest — Golden Rules
1
Set n_jobs=-1. The trees are independent, so using all cores is free speed. The default uses only one core.
2
Set oob_score=True. You get a nearly unbiased estimate of accuracy on new data at almost no extra cost. Check it before you touch the test set.
3
Do not scale features. Tree splits depend only on the order of values, not their size. A StandardScaler changes nothing.
4
Tune max_features first — it controls how different the trees are. Then min_samples_leaf (2–5 for noisy data). For n_estimators, just use enough trees that the OOB error is flat (often 300–500).
5
For imbalanced data, do not judge by accuracy. Use recall, precision, F1, PR-AUC or ROC-AUC. Try class_weight='balanced_subsample', a balanced forest, and — often most useful — a tuned decision threshold.
6
More trees do not cause overfitting. The error falls and then goes flat. Extra trees only cost time and memory. (The trees themselves can still overfit — control that with max_features and min_samples_leaf.)
7
Do not trust MDI importance blindly. Check it with permutation importance on test data (permutation_importance from sklearn.inspection). It is less biased — but correlated features can still share or hide importance, so read both with care.