Precision, Recall & F1 Score

The Story: Dr. Meera's TB Screening Crisis

Why Accuracy Fails on Imbalanced Data

The Confusion Matrix — The Foundation of All Metrics

Every classification metric is derived from a single 2×2 table called the Confusion Matrix. Understanding it is the first step.

Precision — When You Raise the Alarm, How Often Are You Right?

Recall — Of All Real Positives, How Many Did You Find?

The Precision-Recall Trade-off

Precision and Recall pull in opposite directions. As you lower the classification threshold (flag more things as positive), you catch more real positives (recall rises) but also flag more false alarms (precision falls). As you raise the threshold, you become more selective (precision rises) but miss more real positives (recall falls).

F1 Score — The Harmonic Mean of Precision and Recall

Neither Precision nor Recall alone is sufficient. We need a single metric that balances both. The F1 Score does this by taking their harmonic mean — not the arithmetic mean.

F-Beta Score — Tuning the Precision-Recall Balance

The F1 score weights Precision and Recall equally. But in the real world, one is often more important than the other. The F-Beta score lets you control that balance with a single parameter β.

Other Essential Metrics

Specificity — Recall for the Negative Class

All Metrics at a Glance — Using Dr. Meera's Numbers

Advanced Metrics: MCC and Cohen's Kappa

When class imbalance is severe and you need a single metric that's genuinely robust, two metrics stand out above accuracy and F1.

Multi-Class Metrics — Macro, Micro & Weighted Averaging

For problems with more than 2 classes, Precision, Recall, and F1 are computed per class and then averaged. Three averaging strategies exist, each with a different philosophy.

Metric Selection Guide — When to Use What

Python — All Metrics from Scratch

import math

# ── Confusion matrix values ───────────────────────────────────
TP, FP, FN, TN = 45, 100, 5, 850
N_total = TP + FP + FN + TN           # 1000
P = TP + FN                           # all actual positives = 50

# ── Core metrics ─────────────────────────────────────────────
accuracy         = (TP + TN) / N_total
precision        = TP / (TP + FP)
recall           = TP / (TP + FN)
specificity      = TN / (TN + FP)
npv              = TN / (TN + FN)     # Negative Predictive Value
fpr              = FP / (FP + TN)     # False Positive Rate
miss_rate        = FN / (FN + TP)     # False Negative Rate
fdr              = FP / (FP + TP)     # False Discovery Rate

# ── F-scores ─────────────────────────────────────────────────
def f_beta(precision, recall, beta=1):
    b2 = beta ** 2
    if precision + recall == 0:
        return 0.0
    return (1 + b2) * precision * recall / (b2 * precision + recall)

f1 = f_beta(precision, recall, beta=1)
f2 = f_beta(precision, recall, beta=2)
f_half = f_beta(precision, recall, beta=0.5)

# ── Matthews Correlation Coefficient ─────────────────────────
mcc_num  = (TP * TN) - (FP * FN)
mcc_den  = math.sqrt((TP+FP) * (TP+FN) * (TN+FP) * (TN+FN))
mcc      = mcc_num / mcc_den if mcc_den > 0 else 0.0

# ── Balanced Accuracy ─────────────────────────────────────────
balanced_acc = (recall + specificity) / 2

# ── Cohen's Kappa ─────────────────────────────────────────────
p_obs = (TP + TN) / N_total
p_pos = ((TP + FP) / N_total) * ((TP + FN) / N_total)
p_neg = ((TN + FN) / N_total) * ((TN + FP) / N_total)
p_exp = p_pos + p_neg
kappa = (p_obs - p_exp) / (1 - p_exp)

# ── Print results ─────────────────────────────────────────────
print(f"{'Metric':<25} {'Value':>8}")
print("-" * 35)
for name, val in [
    ("Accuracy",         accuracy),
    ("Precision (PPV)",   precision),
    ("Recall (Sensitivity)", recall),
    ("Specificity (TNR)", specificity),
    ("NPV",              npv),
    ("False Positive Rate", fpr),
    ("Miss Rate (FNR)",  miss_rate),
    ("F0.5 Score",       f_half),
    ("F1  Score",        f1),
    ("F2  Score",        f2),
    ("MCC",              mcc),
    ("Balanced Accuracy", balanced_acc),
    ("Cohen's Kappa",    kappa),
]:
    print(f"{name:<25} {val:>8.4f}")

Python — Full Pipeline with scikit-learn

import numpy as np
from sklearn.datasets        import load_breast_cancer
from sklearn.model_selection import train_test_split
from sklearn.preprocessing   import StandardScaler
from sklearn.linear_model    import LogisticRegression
from sklearn.metrics         import (
    confusion_matrix,
    classification_report,
    precision_score, recall_score,
    f1_score, fbeta_score,
    matthews_corrcoef, cohen_kappa_score,
    balanced_accuracy_score,
    precision_recall_fscore_support
)

# ── 1. Load & split ───────────────────────────────────────────
X, y = load_breast_cancer(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42, stratify=y
)
scaler  = StandardScaler()
X_train = scaler.fit_transform(X_train)
X_test  = scaler.transform(X_test)

# ── 2. Train ──────────────────────────────────────────────────
model = LogisticRegression(C=1.0, max_iter=1000, random_state=42)
model.fit(X_train, y_train)
y_pred = model.predict(X_test)

# ── 3. Full classification report ────────────────────────────
print("=== Classification Report ===")
print(classification_report(y_test, y_pred,
      target_names=['Malignant', 'Benign']))

# ── 4. Individual metrics ─────────────────────────────────────
print("=== Individual Metrics (positive class = Benign) ===")
print(f"Precision        : {precision_score(y_test, y_pred):.4f}")
print(f"Recall           : {recall_score(y_test, y_pred):.4f}")
print(f"F1 Score         : {f1_score(y_test, y_pred):.4f}")
print(f"F2 Score         : {fbeta_score(y_test, y_pred, beta=2):.4f}")
print(f"F0.5 Score       : {fbeta_score(y_test, y_pred, beta=0.5):.4f}")
print(f"MCC              : {matthews_corrcoef(y_test, y_pred):.4f}")
print(f"Cohen's Kappa    : {cohen_kappa_score(y_test, y_pred):.4f}")
print(f"Balanced Accuracy: {balanced_accuracy_score(y_test, y_pred):.4f}")

# ── 5. Threshold tuning for high Recall ──────────────────────
y_proba    = model.predict_proba(X_test)[:, 1]
threshold  = 0.3
y_pred_low = (y_proba >= threshold).astype(int)

print(f"\n=== Metrics at threshold = {threshold} ===")
print(f"Precision : {precision_score(y_test, y_pred_low):.4f}")
print(f"Recall    : {recall_score(y_test, y_pred_low):.4f}")
print(f"F1        : {f1_score(y_test, y_pred_low):.4f}")
print(f"F2        : {fbeta_score(y_test, y_pred_low, beta=2):.4f}")

# ── 6. Multi-class macro / weighted / micro ───────────────────
from sklearn.datasets import load_iris
Xi, yi = load_iris(return_X_y=True)
Xtr, Xte, ytr, yte = train_test_split(Xi, yi, test_size=0.25, random_state=0)
mc_model = LogisticRegression(max_iter=500).fit(Xtr, ytr)
yp = mc_model.predict(Xte)

print("\n=== Multi-class Averaging (Iris, 3 classes) ===")
for avg in ['macro', 'weighted', 'micro']:
    p, r, f, _ = precision_recall_fscore_support(yte, yp, average=avg)
    print(f"{avg:>10} → P={p:.3f}  R={r:.3f}  F1={f:.3f}")

print("\n=== Per-Class Breakdown ===")
print(classification_report(yte, yp,
      target_names=['setosa', 'versicolor', 'virginica']))

Golden Rules