Logistic Regression - Binary Classification from Intuition to Math

The Story: Dr. Sharma's Diagnosis Machine

Logistic Regression is the go-to algorithm for binary classification — problems where the output is one of two classes (Yes/No, Spam/Not-Spam, Fraud/Legit, Malignant/Benign). Despite its name, it is a classification algorithm, not regression. It predicts the probability that an observation belongs to a class, then applies a threshold to make the final class decision.

Why Linear Regression Fails for Classification

The Sigmoid Function — The Heart of Logistic Regression

Sigmoid Key Values at a Glance

The Full Logistic Regression Model

The Math Behind It — Log-Odds and the Logit

Why is it called logistic regression? Because it models the log-odds of the outcome as a linear function of the features.

The Cost Function — Binary Cross-Entropy (Log Loss)

Logistic Regression cannot use the same Mean Squared Error cost as Linear Regression — that gives a non-convex surface with many local minima. Instead it uses Binary Cross-Entropy, also called Log Loss.

Gradient Descent — Finding the Best Weights

Unlike Linear Regression, Logistic Regression has no closed-form solution. We minimise J(β) iteratively using Gradient Descent.

The Decision Boundary

Once trained, Logistic Regression draws a linear decision boundary in feature space. Points on one side get Class 0, points on the other get Class 1.

Evaluating the Model — The Confusion Matrix

Accuracy alone is dangerous for classification. If 95% of tumours are benign, a model that always predicts "Benign" gets 95% accuracy — but misses every malignant case. The Confusion Matrix gives the full picture.

Precision, Recall, and F1-Score

ROC Curve and AUC

The ROC Curve (Receiver Operating Characteristic) plots the True Positive Rate (Recall) against the False Positive Rate at every possible threshold from 0 to 1. The area under this curve — AUC — summarises model quality in a single number.

Python — Logistic Regression from Scratch

import numpy as np

# ── Sigmoid function ─────────────────────────────────────────
def sigmoid(z):
    return 1 / (1 + np.exp(-z))

# ── Binary Cross-Entropy Loss ────────────────────────────────
def log_loss(y, p_hat):
    n = len(y)
    eps = 1e-15                           # clip to avoid log(0)
    p_hat = np.clip(p_hat, eps, 1 - eps)
    return -(1 / n) * np.sum(
        y * np.log(p_hat) + (1 - y) * np.log(1 - p_hat)
    )

# ── Training with Gradient Descent ──────────────────────────
def train_logistic(X, y, lr=0.1, epochs=1000):
    n, p  = X.shape
    beta  = np.zeros(p + 1)               # β₀ + p weights
    X_b   = np.column_stack([np.ones(n), X])  # add bias column

    for epoch in range(epochs):
        z      = X_b @ beta                # linear score
        p_hat  = sigmoid(z)              # predicted probability
        error  = p_hat - y                 # residual (p̂ᵢ − yᵢ)
        grad   = (1 / n) * (X_b.T @ error) # gradient vector
        beta  -= lr * grad                 # update weights

        if epoch % 100 == 0:
            loss = log_loss(y, p_hat)
            print(f"Epoch {epoch:4d}  |  Loss: {loss:.4f}")

    return beta

# ── Prediction ───────────────────────────────────────────────
def predict_proba(X, beta):
    X_b = np.column_stack([np.ones(len(X)), X])
    return sigmoid(X_b @ beta)

def predict(X, beta, threshold=0.5):
    return (predict_proba(X, beta) >= threshold).astype(int)

# ── Toy dataset: tumour sizes ────────────────────────────────
np.random.seed(42)
X_train = np.random.randn(100, 2)        # 100 samples, 2 features
y_train = ((X_train[:, 0] + X_train[:, 1]) > 0).astype(float)

beta = train_logistic(X_train, y_train, lr=0.5, epochs=500)
print(f"\nLearned weights: β₀={beta[0]:.3f}  β₁={beta[1]:.3f}  β₂={beta[2]:.3f}")

proba = predict_proba(X_train[:3], beta)
print(f"Predicted probabilities (first 3): {proba.round(3)}")

Python — Full Pipeline with scikit-learn

import numpy as np
from sklearn.datasets         import load_breast_cancer
from sklearn.model_selection  import train_test_split
from sklearn.preprocessing    import StandardScaler
from sklearn.linear_model     import LogisticRegression
from sklearn.metrics          import (
    classification_report, confusion_matrix,
    roc_auc_score, RocCurveDisplay
)

# ── 1. Load Data ─────────────────────────────────────────────
data    = load_breast_cancer()        # Wisconsin Breast Cancer dataset
X, y    = data.data, data.target       # 569 samples, 30 features

# ── 2. Train / Test Split ────────────────────────────────────
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42, stratify=y
)

# ── 3. Feature Scaling (critical for Logistic Regression) ───
scaler  = StandardScaler()
X_train = scaler.fit_transform(X_train)
X_test  = scaler.transform(X_test)     # transform only — no fit!

# ── 4. Train Model ───────────────────────────────────────────
model = LogisticRegression(
    C=1.0,             # inverse of regularisation strength (1/λ)
    solver='lbfgs',   # efficient solver for small-medium datasets
    max_iter=1000,
    random_state=42
)
model.fit(X_train, y_train)

# ── 5. Evaluate ──────────────────────────────────────────────
y_pred       = model.predict(X_test)
y_proba      = model.predict_proba(X_test)[:, 1]  # P(Class=1)

print("Confusion Matrix:")
print(confusion_matrix(y_test, y_pred))

print("\nClassification Report:")
print(classification_report(y_test, y_pred, target_names=data.target_names))

auc = roc_auc_score(y_test, y_proba)
print(f"ROC-AUC Score: {auc:.4f}")

# ── 6. Tune threshold for high Recall ───────────────────────
threshold = 0.3
y_pred_low = (y_proba >= threshold).astype(int)
print(f"\nAt threshold={threshold}:")
print(classification_report(y_test, y_pred_low, target_names=data.target_names))

Regularisation — Preventing Overfitting

When there are many features (especially more features than samples), Logistic Regression can overfit — it memorises training noise. Regularisation adds a penalty term to the cost function to keep weights small.

Multiclass Logistic Regression

Standard Logistic Regression is binary. Two strategies extend it to multiple classes:

from sklearn.linear_model import LogisticRegression
from sklearn.datasets     import load_iris
from sklearn.preprocessing import StandardScaler
from sklearn.model_selection import train_test_split

# ── Iris: 3 classes ──────────────────────────────────────────
X, y = load_iris(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42
)

scaler  = StandardScaler()
X_train = scaler.fit_transform(X_train)
X_test  = scaler.transform(X_test)

# ── Multinomial (Softmax) Logistic Regression ────────────────
model = LogisticRegression(
    multi_class='multinomial',
    solver='lbfgs',
    C=1.0,
    max_iter=500
)
model.fit(X_train, y_train)

print(f"Test Accuracy : {model.score(X_test, y_test):.4f}")  # ~0.967

# Probability distribution for first test sample
print("Class probabilities (sample 1):", model.predict_proba(X_test[:1]).round(3))
# [[0.001  0.072  0.927]]  → Class 2 (Virginica) with 92.7% confidence

Assumptions of Logistic Regression

Golden Rules