Linear Regression

The Story: Priya Wants to Price Her Flat

Linear Regression is the most fundamental supervised machine-learning algorithm. It finds the straight line that best describes the relationship between one or more input variables (features) and a continuous output variable (target). Once the line is found, predicting new values is as simple as plugging a number into a formula.

Visualising the Idea

Priya plots her data on a graph. Each dot is a flat — x-axis = size, y-axis = price. The dots form a rough upward trend. Linear Regression draws the best-fit line through those dots.

The Equation of a Straight Line

In school you learned y = mx + c. Linear Regression uses the same idea, just with machine-learning notation:

The Math: How the Best Line is Found (OLS)

"Best fit" is not guesswork. The algorithm minimises the Sum of Squared Residuals (SSR) — also called the Residual Sum of Squares (RSS) or Sum of Squared Errors (SSE). A residual is the gap between a real data point and the line's prediction.

The Cost Function

We want to minimise the Loss function L:

Differential Equations — Finding the Minimum

To minimise L, take the partial derivative with respect to each parameter and set it to zero. This is where calculus meets machine learning.

Numerical Example — Priya's 5 Flats

Let's use a small dataset so the math is visible. Priya takes 5 flats:

RMSE — Root Mean Squared Error

Now that Priya has a model, she needs to know: how wrong is it, on average? The most common metric for this is RMSE.

Step-by-Step RMSE for Priya's 5 Flats

R² — The Coefficient of Determination

RMSE tells you the absolute error in real units. But it does not tell you how good your model is compared to having no model at all. R² answers that question.

Calculating R² for Priya's Data

Adjusted R² — The Honest Version

Adjusted R² corrects for the "adding features always helps" bias in R². It penalises you for adding features that do not genuinely improve the model. It only increases if a new feature adds more than it costs.

How the Penalty Works

Numerical Example — Adding a Useless Feature

Priya adds a second feature: the flat owner's lucky number (random noise).

Metrics at a Glance

Python Implementation

Manual OLS from Scratch

import math

x = [600, 800, 1000, 1200, 1400]    # size (sq ft)
y = [65,  85,  105,  120,  145]     # price (lakhs)

n     = len(x)
x_bar = sum(x) / n                   # 1000.0
y_bar = sum(y) / n                   # 104.0

# OLS coefficients
num   = sum((xi - x_bar) * (yi - y_bar) for xi, yi in zip(x, y))  # 39000
denom = sum((xi - x_bar) ** 2        for xi     in x)             # 400000

beta1 = num / denom                  # slope     = 0.0975
beta0 = y_bar - beta1 * x_bar       # intercept = 6.5

print(f"Intercept (β₀): {beta0:.4f}")  # 6.5
print(f"Slope     (β₁): {beta1:.4f}")  # 0.0975

# Predictions
y_pred = [beta0 + beta1 * xi for xi in x]
print("Predictions:", [round(p, 2) for p in y_pred])
# [65.0, 84.5, 104.0, 123.5, 143.0]

Computing RMSE, R², Adjusted R²

import math

def rmse(y_true, y_pred):
    n   = len(y_true)
    sse = sum((yt - yp) ** 2 for yt, yp in zip(y_true, y_pred))
    return math.sqrt(sse / n)

def r_squared(y_true, y_pred):
    y_bar  = sum(y_true) / len(y_true)
    ss_tot = sum((yt - y_bar) ** 2 for yt     in y_true)
    ss_res = sum((yt - yp)    ** 2 for yt, yp in zip(y_true, y_pred))
    return 1 - (ss_res / ss_tot)

def adjusted_r2(y_true, y_pred, p):
    n  = len(y_true)
    r2 = r_squared(y_true, y_pred)
    return 1 - (1 - r2) * (n - 1) / (n - p - 1)

y_true = [65, 85, 105, 120, 145]
y_pred = [65.0, 84.5, 104.0, 123.5, 143.0]

print(f"RMSE        : {rmse(y_true, y_pred):.4f}")             # 1.8708
print(f"R²          : {r_squared(y_true, y_pred):.4f}")         # 0.9954
print(f"Adjusted R² : {adjusted_r2(y_true, y_pred, p=1):.4f}")  # 0.9939

Using scikit-learn

import numpy as np
from sklearn.linear_model  import LinearRegression
from sklearn.metrics       import mean_squared_error, r2_score

X = np.array([[600], [800], [1000], [1200], [1400]])  # 2-D for sklearn
y = np.array([65, 85, 105, 120, 145])

model = LinearRegression()
model.fit(X, y)

print(f"Intercept β₀ : {model.intercept_:.4f}")   # 6.5
print(f"Slope     β₁ : {model.coef_[0]:.4f}")      # 0.0975

y_pred = model.predict(X)

mse  = mean_squared_error(y, y_pred)
rmse = np.sqrt(mse)
r2   = r2_score(y, y_pred)

n, p   = len(y), X.shape[1]
adj_r2 = 1 - (1 - r2) * (n - 1) / (n - p - 1)

print(f"RMSE         : {rmse:.4f}")     # 1.8708
print(f"R²           : {r2:.4f}")       # 0.9954
print(f"Adjusted R²  : {adj_r2:.4f}")  # 0.9939

# Predict the new flat
new_flat = np.array([[1200]])
print(f"Predicted price for 1200 sq ft: {model.predict(new_flat)[0]:.1f} L")
# 123.5 L

Assumptions of Linear Regression

Linear Regression is not magic — it comes with four assumptions that must hold for the model and its metrics to be trustworthy.

Golden Rules