Random Forest

The Story That Explains Random Forest

A Random Forest is an ensemble of many decision trees, each trained on a slightly different random slice of your data and a random subset of features. When predicting, every tree votes, and the majority wins (classification) or the average is taken (regression). No single tree needs to be perfect — together they are.

The Foundation — Decision Trees (Quick Recap)

Before building a forest, you need to understand a single tree. A decision tree splits your data by asking questions — one feature at a time — until it reaches a prediction.

The Two Ingredients of Random Forest

Random Forest solves overfitting by injecting randomness in two separate ways. Both are essential — neither alone is enough.

How Bootstrap Sampling Works

Visual Diagram — Inside a Random Forest

Classification vs Regression

Python Implementation

Classification — Predicting Customer Churn

from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import train_test_split
from sklearn.metrics import classification_report
from sklearn.datasets import make_classification
import numpy as np

# Simulated dataset: 5000 customers, 15 features
X, y = make_classification(
    n_samples=5000, n_features=15,
    n_informative=10, n_redundant=3,
    random_state=42
)

X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42
)

# Build the Random Forest
rf = RandomForestClassifier(
    n_estimators=300,        # number of trees
    max_features='sqrt',     # features per split -> sqrt(15) ~ 4
    max_depth=None,          # grow trees fully
    min_samples_leaf=1,      # minimum samples at a leaf
    bootstrap=True,          # enable bagging
    oob_score=True,          # free OOB validation
    n_jobs=-1,               # use all CPU cores
    random_state=42
)

rf.fit(X_train, y_train)

print(f"OOB Score (free accuracy estimate): {rf.oob_score_:.4f}")
print(f"Test Accuracy:                      {rf.score(X_test, y_test):.4f}")

y_pred = rf.predict(X_test)
print(classification_report(y_test, y_pred))

Regression — Predicting House Prices

from sklearn.ensemble import RandomForestRegressor
from sklearn.datasets import fetch_california_housing
from sklearn.model_selection import train_test_split
from sklearn.metrics import mean_absolute_error, r2_score

housing = fetch_california_housing()
X, y = housing.data, housing.target

X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42
)

rf_reg = RandomForestRegressor(
    n_estimators=300,
    max_features='sqrt',     # or 1/3 of features
    oob_score=True,
    n_jobs=-1,
    random_state=42
)

rf_reg.fit(X_train, y_train)
y_pred = rf_reg.predict(X_test)

print(f"R2 Score: {r2_score(y_test, y_pred):.4f}")
print(f"MAE:      ${mean_absolute_error(y_test, y_pred)*100_000:.0f}")

Key Hyperparameters

Hyperparameter Tuning with RandomizedSearchCV

from sklearn.model_selection import RandomizedSearchCV
from scipy.stats import randint

param_dist = {
    'n_estimators':      randint(100, 600),
    'max_features':      ['sqrt', 'log2', 0.3, 0.5],
    'max_depth':         [None, 10, 20, 30, 50],
    'min_samples_leaf':  randint(1, 10),
    'min_samples_split': randint(2, 15),
}

search = RandomizedSearchCV(
    RandomForestClassifier(n_jobs=-1, oob_score=True, random_state=42),
    param_distributions=param_dist,
    n_iter=50,           # try 50 random combinations
    cv=5,
    scoring='f1_weighted',
    n_jobs=-1,
    random_state=42
)

search.fit(X_train, y_train)
print("Best params:", search.best_params_)
print("Best CV F1: ", search.best_score_)

Feature Importance

One of Random Forest's greatest practical advantages is its built-in ability to rank which features drive predictions. This is called Mean Decrease in Impurity (MDI) — it measures how much each feature reduces Gini impurity (or MSE for regression) across all trees, on average.

import pandas as pd
import matplotlib.pyplot as plt

feature_names = housing.feature_names  # ['MedInc', 'HouseAge', 'AveRooms', ...]

importances = rf_reg.feature_importances_
std         = np.std([t.feature_importances_ for t in rf_reg.estimators_], axis=0)

feat_df = pd.DataFrame({
    'feature':    feature_names,
    'importance': importances,
    'std':        std
}).sort_values('importance', ascending=False)

print(feat_df.to_string(index=False))

# Plot
feat_df.plot.barh(x='feature', y='importance', xerr='std', legend=False)
plt.xlabel('Mean Decrease in Impurity')
plt.title('Random Forest Feature Importance')
plt.tight_layout()
plt.show()

Permutation Importance (More Reliable)

from sklearn.inspection import permutation_importance

result = permutation_importance(
    rf_reg, X_test, y_test,
    n_repeats=10,
    n_jobs=-1,
    random_state=42
)

perm_df = pd.DataFrame({
    'feature':    feature_names,
    'importance': result.importances_mean,
    'std':        result.importances_std
}).sort_values('importance', ascending=False)

print(perm_df.to_string(index=False))

Bias–Variance Tradeoff: Single Tree vs Forest

When to Use Random Forest

Handling Class Imbalance

Random Forest can struggle with heavily imbalanced datasets (e.g. fraud detection: 99% normal, 1% fraud). The forest majority-votes, so rare classes get drowned out. Here are your options:

from imblearn.ensemble import BalancedRandomForestClassifier

brf = BalancedRandomForestClassifier(
    n_estimators=300,
    sampling_strategy='auto',  # undersample majority to match minority
    replacement=True,
    n_jobs=-1,
    random_state=42
)
brf.fit(X_train, y_train)

# Or with class_weight in standard sklearn:
rf_balanced = RandomForestClassifier(
    n_estimators=300,
    class_weight='balanced_subsample',
    n_jobs=-1,
    random_state=42
)
rf_balanced.fit(X_train, y_train)

A Complete Real-World Example — Titanic Survival

import pandas as pd
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import cross_val_score
from sklearn.preprocessing import LabelEncoder

# Load data
df = pd.read_csv('titanic.csv')

# Simple preprocessing
df['Age'].fillna(df['Age'].median(), inplace=True)
df['Embarked'].fillna('S', inplace=True)
df['Sex'] = LabelEncoder().fit_transform(df['Sex'])
df['Embarked'] = LabelEncoder().fit_transform(df['Embarked'])

features = ['Pclass', 'Sex', 'Age', 'SibSp', 'Parch', 'Fare', 'Embarked']
X = df[features]
y = df['Survived']

# Note: Random Forest needs NO feature scaling
rf = RandomForestClassifier(
    n_estimators=500,
    max_features='sqrt',
    max_depth=None,
    min_samples_leaf=2,
    class_weight='balanced',
    oob_score=True,
    n_jobs=-1,
    random_state=42
)

# 5-fold cross-validation
cv_scores = cross_val_score(rf, X, y, cv=5, scoring='accuracy')
print(f"CV Accuracy: {cv_scores.mean():.4f} +/- {cv_scores.std():.4f}")

# Fit and check OOB
rf.fit(X, y)
print(f"OOB Score:   {rf.oob_score_:.4f}")

# Feature importance
for feat, imp in sorted(zip(features, rf.feature_importances_),
                        key=lambda x: -x[1]):
    print(f"  {feat:12s}: {imp:.4f}")

Random Forest vs Gradient Boosting — Quick Comparison

Golden Rules