ROC Curve & AUC

The Story: Aisha's Three Fraud Models

What Is the ROC Curve?

The ROC Curve (Receiver Operating Characteristic Curve) is a graph that shows how a classifier performs at every possible classification threshold — from 0.0 (classify everything as fraud) to 1.0 (classify nothing as fraud).

At each threshold, the model produces a different confusion matrix. The ROC curve plots two numbers extracted from that matrix:

Building the ROC Curve Step by Step

Let's use a tiny dataset — 10 transactions, 5 fraudulent (F) and 5 legitimate (L). A model has assigned each a fraud probability. We sort by probability descending and sweep the threshold:

Total Positives (Fraud): P = 5  |  Total Negatives (Legit): N = 5
At each rank boundary we record the cumulative TP, FP, and compute TPR = TP/5, FPR = FP/5:

The ROC Curve — Visualised

Calculating AUC — The Trapezoidal Rule

AUC is the area under the ROC curve, computed using the trapezoidal rule: for each consecutive pair of points (FPR₁, TPR₁) and (FPR₂, TPR₂), the area of the trapezoid beneath them is:

Multi-Model Comparison — The Real Power of ROC

The true strength of ROC curves is comparing multiple models simultaneously, across all thresholds at once. Aisha plots all three of her models on the same axes:

Interpreting the AUC Score

ROC-AUC vs Precision-Recall AUC — When Each Wins

The ROC curve is not always the right tool. On severely imbalanced datasets — like fraud (0.1% positive rate) — the ROC curve can be misleadingly optimistic. The Precision-Recall (PR) curve is more honest in those cases.

Choosing the Right Operating Point

The ROC curve shows all possible thresholds. Choosing which point to operate at requires a business decision. Three common strategies:

Multi-Class ROC — One-vs-Rest (OvR)

For problems with more than two classes (e.g. classifying a tumour as Benign, Malignant Type A, or Malignant Type B), the standard binary ROC cannot be directly applied. The most common extension is One-vs-Rest (OvR):

Python — ROC Curve & AUC from Scratch

import numpy as np

# ── Build ROC curve manually ─────────────────────────────────
def roc_curve_manual(y_true, y_scores):
    """Compute TPR and FPR at every unique threshold."""
    thresholds = np.sort(np.unique(y_scores))[::-1]  # descending
    P = np.sum(y_true == 1)                       # total positives
    N = np.sum(y_true == 0)                       # total negatives

    tpr_list = [0.0]
    fpr_list = [0.0]

    for thresh in thresholds:
        y_pred = (y_scores >= thresh).astype(int)
        TP = np.sum((y_pred == 1) & (y_true == 1))
        FP = np.sum((y_pred == 1) & (y_true == 0))
        tpr_list.append(TP / P)
        fpr_list.append(FP / N)

    tpr_list.append(1.0)
    fpr_list.append(1.0)
    return np.array(fpr_list), np.array(tpr_list)

# ── AUC via trapezoidal rule ─────────────────────────────────
def auc_trapezoid(fpr, tpr):
    """Area under the ROC curve using np.trapz."""
    return np.trapz(tpr, fpr)   # integrates TPR over FPR

# ── 10-transaction example ───────────────────────────────────
y_true   = np.array([1, 0, 1, 1, 0, 0, 1, 0, 1, 0])
y_scores = np.array([0.95, 0.88, 0.82, 0.71, 0.60,
                      0.45, 0.38, 0.25, 0.15, 0.05])

fpr, tpr = roc_curve_manual(y_true, y_scores)
auc      = auc_trapezoid(fpr, tpr)

print("FPR points:", fpr.round(2))
print("TPR points:", tpr.round(2))
print(f"AUC         : {auc:.4f}")      # 0.6400

# ── Youden's J — find optimal threshold ──────────────────────
thresholds = np.sort(np.unique(y_scores))[::-1]
j_scores   = tpr[1:-1] - fpr[1:-1]     # TPR - FPR at each threshold
best_idx   = np.argmax(j_scores)
best_thresh = thresholds[best_idx]

print(f"Best threshold (Youden's J): {best_thresh:.2f}")
print(f"  TPR at best: {tpr[best_idx+1]:.2f}  "
      f"FPR at best: {fpr[best_idx+1]:.2f}")

Python — Full Pipeline with scikit-learn

import numpy as np
import matplotlib.pyplot as plt
from sklearn.datasets        import make_classification
from sklearn.model_selection import train_test_split, cross_val_score
from sklearn.preprocessing   import StandardScaler
from sklearn.linear_model    import LogisticRegression
from sklearn.ensemble        import RandomForestClassifier, GradientBoostingClassifier
from sklearn.metrics         import (
    roc_curve, roc_auc_score,
    precision_recall_curve, average_precision_score,
    RocCurveDisplay
)

# ── 1. Create imbalanced dataset ─────────────────────────────
X, y = make_classification(
    n_samples=5000, n_features=20,
    weights=[0.95, 0.05],         # 95% legit, 5% fraud
    random_state=42
)

X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.25, random_state=42, stratify=y
)

scaler  = StandardScaler()
X_train = scaler.fit_transform(X_train)
X_test  = scaler.transform(X_test)

# ── 2. Train three models ────────────────────────────────────
models = {
    'Logistic Regression'    : LogisticRegression(C=1.0, max_iter=500),
    'Random Forest'          : RandomForestClassifier(n_estimators=100, random_state=42),
    'Gradient Boosting'      : GradientBoostingClassifier(n_estimators=100, random_state=42),
}

for name, model in models.items():
    model.fit(X_train, y_train)

# ── 3. Compute & plot ROC curves ─────────────────────────────
fig, axes = plt.subplots(1, 2, figsize=(14, 6))

colors = ['#6366f1', '#f59e0b', '#34d399']

for (name, model), color in zip(models.items(), colors):
    y_proba = model.predict_proba(X_test)[:, 1]

    # ROC
    fpr, tpr, _ = roc_curve(y_test, y_proba)
    auc          = roc_auc_score(y_test, y_proba)
    axes[0].plot(fpr, tpr, color=color, lw=2,
                 label=f"{name} (AUC={auc:.3f})")

    # Precision-Recall
    prec, rec, _ = precision_recall_curve(y_test, y_proba)
    ap            = average_precision_score(y_test, y_proba)
    axes[1].plot(rec, prec, color=color, lw=2,
                 label=f"{name} (AP={ap:.3f})")

# ROC — diagonal baseline
axes[0].plot([0, 1], [0, 1], 'k--', label='Random (AUC=0.50)')
axes[0].set_xlabel('False Positive Rate')
axes[0].set_ylabel('True Positive Rate')
axes[0].set_title('ROC Curve')
axes[0].legend()

# PR — baseline (always-positive classifier)
baseline = y_test.sum() / len(y_test)
axes[1].axhline(y=baseline, color='k', linestyle='--',
                label=f'Random (AP={baseline:.3f})')
axes[1].set_xlabel('Recall')
axes[1].set_ylabel('Precision')
axes[1].set_title('Precision-Recall Curve')
axes[1].legend()

plt.tight_layout()
plt.show()

# ── 4. Optimal threshold via Youden's J ──────────────────────
gb_proba     = models['Gradient Boosting'].predict_proba(X_test)[:, 1]
fpr, tpr, thresholds = roc_curve(y_test, gb_proba)

j_scores   = tpr - fpr
best_idx   = np.argmax(j_scores)
best_thresh = thresholds[best_idx]
print(f"Optimal threshold (Youden): {best_thresh:.4f}")
print(f"  TPR = {tpr[best_idx]:.4f}   FPR = {fpr[best_idx]:.4f}")

# ── 5. Multi-class AUC (if needed) ───────────────────────────
from sklearn.datasets   import load_iris
from sklearn.multiclass import OneVsRestClassifier

X_mc, y_mc = load_iris(return_X_y=True)
X_tr, X_te, y_tr, y_te = train_test_split(X_mc, y_mc, test_size=0.3)

ovr_model = OneVsRestClassifier(LogisticRegression(max_iter=500))
ovr_model.fit(X_tr, y_tr)
y_proba_mc = ovr_model.predict_proba(X_te)

macro_auc = roc_auc_score(y_te, y_proba_mc,
                            multi_class='ovr', average='macro')
print(f"Multi-class Macro AUC (Iris): {macro_auc:.4f}")

Common Mistakes with ROC & AUC

Golden Rules