Entropy, Information Gain and Gini Impurity

The Story: Why "Impurity" Matters

What Is Impurity? — The Core Concept

Before we touch any formula, let's build the intuition. A node in a decision tree contains a group of training samples. Impurity measures how mixed the class labels are in that group.

Entropy — Disorder Measured in Bits

The Entropy Curve — Visualised

Step-by-Step Entropy Calculation — Weather Dataset

We have 14 days of weather data. The target is Play Tennis? The full set S contains 9 Yes and 5 No.

Entropy for Multiple Classes

Entropy works for any number of classes — not just binary. The formula is identical; you just sum more terms. The maximum possible entropy scales with the number of classes.

Information Gain — Measuring Split Quality

Information Gain answers the question: "How much does asking this question reduce my uncertainty about the answer?" It is simply the difference in entropy before and after the split, weighted by the size of each child group.

Worked Example: Which Attribute Splits the Weather Data Best?

Parent entropy H(S) = 0.940 bits. Now we calculate IG for each attribute.

Gini Impurity — The CART Algorithm's Measure

Gini Values Across Class Distributions

Gini Calculation — Same Weather Dataset

Gini for Multiple Classes — Maximum Values

Just like entropy, the maximum Gini Impurity increases with the number of classes. The formula for maximum Gini is (n−1)/n — occurring when all classes are equally likely.

Entropy vs Gini — Head-to-Head

The Bias Problem — and Gain Ratio (C4.5)

Python Implementation — From Scratch and with sklearn

Manual Calculation — Entropy, Gini and Information Gain

import numpy as np

# ── Helper functions ──────────────────────────────────────────
def entropy(labels):
    """Shannon entropy of a list of class labels."""
    n = len(labels)
    if n == 0:
        return 0.0
    classes, counts = np.unique(labels, return_counts=True)
    probs = counts / n
    # Convention: 0 * log2(0) = 0
    return -np.sum(probs * np.log2(probs + 1e-12))

def gini(labels):
    """Gini impurity of a list of class labels."""
    n = len(labels)
    if n == 0:
        return 0.0
    _, counts = np.unique(labels, return_counts=True)
    probs = counts / n
    return 1 - np.sum(probs ** 2)

def information_gain(parent_labels, child_groups):
    """IG = H(parent) - weighted sum of H(children)."""
    n_parent = len(parent_labels)
    h_parent = entropy(parent_labels)
    weighted_child = sum(
        (len(group) / n_parent) * entropy(group)
        for group in child_groups
    )
    return h_parent - weighted_child

def gini_gain(parent_labels, child_groups):
    """Gini Gain = Gini(parent) - weighted Gini(children)."""
    n_parent = len(parent_labels)
    g_parent = gini(parent_labels)
    weighted_child = sum(
        (len(group) / n_parent) * gini(group)
        for group in child_groups
    )
    return g_parent - weighted_child

# ── Play Tennis Dataset ───────────────────────────────────────
y = np.array([0,0,1,1,1,0,1,0,1,1,1,1,1,0])  # 0=No, 1=Yes

# Outlook split: Sunny=[0,1,2,7,10] → indices → labels
sunny    = y[[0,1,7,8,10]]   # D1,D2,D8,D9,D11
overcast = y[[2,6,11,12]]    # D3,D7,D12,D13
rain     = y[[3,4,5,9,13]]   # D4,D5,D6,D10,D14

print("=== ENTROPY ===")
print(f"H(parent)  = {entropy(y):.4f}")
print(f"H(Sunny)   = {entropy(sunny):.4f}")
print(f"H(Overcast)= {entropy(overcast):.4f}")
print(f"H(Rain)    = {entropy(rain):.4f}")
print(f"IG(Outlook)= {information_gain(y, [sunny, overcast, rain]):.4f}")

print("\n=== GINI ===")
print(f"Gini(parent)  = {gini(y):.4f}")
print(f"Gini(Sunny)   = {gini(sunny):.4f}")
print(f"Gini(Overcast)= {gini(overcast):.4f}")
print(f"Gini(Rain)    = {gini(rain):.4f}")
print(f"Gini Gain(Outlook) = {gini_gain(y, [sunny, overcast, rain]):.4f}")

Compare All Attributes at Once

import pandas as pd

# Full dataset as a DataFrame
df = pd.DataFrame({
    'Outlook' : ['Sunny','Sunny','Overcast','Rain','Rain','Rain',
                 'Overcast','Sunny','Sunny','Rain','Sunny',
                 'Overcast','Overcast','Rain'],
    'Temp'    : ['Hot','Hot','Hot','Mild','Cool','Cool','Cool',
                 'Mild','Cool','Mild','Mild','Mild','Hot','Mild'],
    'Humidity': ['High','High','High','High','Normal','Normal','Normal',
                 'High','Normal','Normal','Normal','High','Normal','High'],
    'Wind'    : ['Weak','Strong','Weak','Weak','Weak','Strong','Strong',
                 'Weak','Weak','Weak','Strong','Strong','Weak','Strong'],
    'Play'    : [0,0,1,1,1,0,1,0,1,1,1,1,1,0]
})

results = []
for col in ['Outlook', 'Temp', 'Humidity', 'Wind']:
    groups = [df[df[col] == v]['Play'].values for v in df[col].unique()]
    ig = information_gain(df['Play'].values, groups)
    gg = gini_gain(df['Play'].values, groups)
    results.append({'Attribute': col, 'Info Gain': ig, 'Gini Gain': gg})

summary = pd.DataFrame(results).sort_values('Info Gain', ascending=False)
print(summary.to_string(index=False, float_format='{:.4f}'.format))

sklearn: Switching Between Gini and Entropy

from sklearn.tree import DecisionTreeClassifier, export_text
from sklearn.preprocessing import OrdinalEncoder
import numpy as np

X_raw = df[['Outlook','Temp','Humidity','Wind']].values
y     = df['Play'].values

enc = OrdinalEncoder()
X   = enc.fit_transform(X_raw)

for criterion in ['gini', 'entropy']:
    dt = DecisionTreeClassifier(criterion=criterion, random_state=42)
    dt.fit(X, y)
    print(f"\n=== criterion='{criterion}' ===")
    print(export_text(dt, feature_names=['Outlook','Temp','Humidity','Wind']))
    print("Feature importances:")
    for f, imp in zip(['Outlook','Temp','Humidity','Wind'], dt.feature_importances_):
        print(f"  {f:<12} {imp:.4f}")

Side-by-Side Split Quality Diagram

The diagram below shows the same parent node being split two ways. Split A is good — children are purer. Split B is poor — children are as mixed as the parent.

Common Pitfalls and How to Avoid Them

Golden Rules