⚖️

Data Transformation — Normalisation & Standardisation

The bridge between raw data and learnable patterns. Put every feature on a fair scale, fix skewed shapes, encode text, and wrap it all in a leak-proof pipeline.

Scaling Distribution Encoding Pipelines

Press Next → or use ← → arrow keys

Section 01

Why Transformation Is Essential

🏦 REAL STORY · 71% → 88%

The model that only learned from salary

A Bengaluru fintech built a KNN loan-default model on age (18–65), employment years (0–40), dependents (0–8) and salary (₹2L–₹50L). Because salary's raw numbers were thousands of times larger, it owned 98% of every distance calculation — the other features were noise. One StandardScaler lifted accuracy from 71% to 88%.

Section 01 · Sensitivity

Which Algorithms Actually Need Scaling?

needs it

📏 Distance-based

KNN, SVM, K-Means measure distance — a big-range feature drowns the rest.

needs it

📉 Gradient-based

Linear/Logistic Regression & Neural Nets converge faster on comparable scales.

needs it

➗ Regularised

Ridge, Lasso, ElasticNet penalise coefficients — unfair unless features share a scale.

skip it

🌳 Tree-based

Decision Trees, Random Forest, XGBoost split on thresholds — scale-invariant by design.

needs it

🧭 PCA

Variance-driven — an unscaled large feature hijacks the principal components.

check skew

📊 Any model

If |skew| > 1, reshape the distribution before scaling — scaling alone won't fix it.

Section 02

The Core Four Scaling Methods

📏 Min-Max

x' = (x − min) / (max − min)

Squeezes into [0,1], keeps shape. Great for neural nets — but outlier-sensitive.

📐 Z-Score (Standard)

x' = (x − μ) / σ

Centres at mean 0, std 1. The default for linear models, PCA, SVM.

🛡️ Robust

x' = (x − median) / IQR

Uses median & IQR — outliers can't distort it. For legitimate extremes.

📉 Log

x' = log(1 + x)

Compresses right-skew. Changes shape, not just range. Use log1p for zeros.

➕

Two more in the toolbox

MaxAbs x/|max| → [−1,1], ideal for sparse TF-IDF. Power transforms (Box-Cox / Yeo-Johnson) automatically search for the shape closest to normal.

Section 03

Min-Max Normalisation → [0, 1]

A medical-imaging net mixed pixel intensity (0–255), blood pressure (60–180) and a 0–1 flag. Gradients exploded — loss oscillated for 200 epochs. After Min-Max it converged cleanly in 40.

from sklearn.preprocessing import MinMaxScaler
scaler = MinMaxScaler(feature_range=(0, 1))
scaler.fit(X_train[num_cols])              # learn min/max from TRAIN only
X_train[num_cols] = scaler.transform(X_train[num_cols])
X_test[num_cols]  = scaler.transform(X_test[num_cols])   # reuse train's min/max

Section 04

Z-Score Standardisation → μ=0, σ=1

A bank read its loan model's salary coefficient as 0.000002 vs credit score 1.4 and "concluded" credit score mattered 700,000× more. Pure scale illusion — after standardising, salary became the strongest predictor and coefficients were finally comparable.

from sklearn.preprocessing import StandardScaler
scaler = StandardScaler()
scaler.fit(X_train[num_cols])
X_train[num_cols] = scaler.transform(X_train[num_cols])
X_test[num_cols]  = scaler.transform(X_test[num_cols])
scaler.mean_, np.sqrt(scaler.var_)      # learned params for inference

Section 05

Robust Scaling & the Full Comparison

Scaler	Formula	Range	Outlier-safe	Best for
MinMax	(x−min)/(max−min)	[0, 1]	no	Neural nets, image data
Standard	(x−μ)/σ	(−∞, ∞)	partly	Linear, PCA, SVM, logistic
Robust	(x−median)/IQR	(−∞, ∞)	yes	Legitimate extreme values
MaxAbs	x/\|max\|	[−1, 1]	no	Sparse matrices, TF-IDF

from sklearn.preprocessing import RobustScaler
scaler = RobustScaler(quantile_range=(25.0, 75.0))   # median + IQR
X_train[num_cols] = scaler.fit(X_train[num_cols]).transform(X_train[num_cols])
# outliers still exist — they just no longer distort everyone else's scaling

Section 05 · Effect

One Outlier, Three Very Different Results

🛡️

Same data, same outlier

MinMax pins the range to the outlier and squashes every normal point into a corner. Standard is dragged less but still distorted. Robust uses median & IQR, so the bulk keeps its spread and the outlier simply sits far away.

Section 06

Log & Power Transforms — Fixing Skew

An insurer's linear model kept underpricing corporate clients: premiums ran ₹5k–₹15k for most, ₹20L+ for a few. Skew 4.8. After np.log1p(), skew fell to 0.3 and RMSE on high-value customers improved 62%.

Section 06 · Code

Choosing a Transform & Checking Skew

# log — the practical default
df['amt_log'] = np.log1p(df['amt'])

from sklearn.preprocessing import \
    PowerTransformer, QuantileTransformer

# Yeo-Johnson — handles negatives
pt = PowerTransformer('yeo-johnson',
                      standardize=True)
df[['amt_yj']] = pt.fit_transform(df[['amt']])

# Quantile — forces a normal shape
qt = QuantileTransformer(
     output_distribution='normal')
df[['amt_qt']] = qt.fit_transform(df[['amt']])

df['amt'].skew()   # diagnose first!

PICK BY SKEW & SIGN

log1pright-skew, values ≥ 0

sqrtmoderate right-skew

box-coxstrictly positive, auto-λ

yeo-jhas zeros / negatives

quantileforce normal, big data

🔍

Scaling ≠ reshaping

A scaler rescales but keeps the skew. If |skew| > 1, transform the shape first, then scale.

Section 07

Categorical Encoding — Text to Numbers

Method	When	Output	Risk
One-Hot	Nominal, < 15 values (city, gender)	n binary columns	low
Ordinal	Ordered (Low < Med < High)	1 col (0,1,2…)	low
Label	Binary, or tree models only	1 col	implies order
Target	High cardinality (>15)	1 col (mean target)	leakage
Frequency	High cardinality, leak-safe	1 col (count %)	low

pd.get_dummies(df, columns=['city','gender'], dtype=int)          # 1 · one-hot
OrdinalEncoder(categories=[['Low','Medium','High']])            # 2 · ordinal
df['city_tgt']  = df['city'].map(df.groupby('city')['target'].mean())  # 4 · target
df['city_freq'] = df['city'].map(df['city'].value_counts()/len(df)) # 5 · frequency

Section 07 · Pitfall

The False-Ordering Trap

⚠️

The most common encoding mistake

Label-encoding unordered categories for a linear model or neural net tells it Chennai is "greater than" Delhi. Use one-hot for nominal categories — reserve label encoding for binary columns or tree models.

Section 08

Feature Engineering — Creating New Signals

# bin continuous → categorical
df['age_group'] = pd.cut(df['age'],
  bins=[0,25,35,45,55,100],
  labels=['18-25','26-35','36-45','46-55','55+'])

# interaction ratios
df['spend_per_day'] = df['amt']/(df['days']+1)

# date features
df['dow']     = df['order_date'].dt.dayofweek
df['is_wknd'] = df['dow'] >= 5

from sklearn.preprocessing import PolynomialFeatures
poly = PolynomialFeatures(degree=2)   # x, x², x·y…

💡

Ratios and flags surface hidden signal

spend_per_day and is_weekend often out-rank the raw columns they came from — the pattern was there, just not exposed.

Section 09

One Leak-Proof Transformation Pipeline

num = Pipeline([('imp', SimpleImputer('median')),
                ('sc',  StandardScaler())])
nom = Pipeline([('imp', SimpleImputer('most_frequent')),
                ('oh',  OneHotEncoder(
                          handle_unknown='ignore'))])
ordp = Pipeline([('imp', SimpleImputer('most_frequent')),
                 ('ord', OrdinalEncoder(
                           categories=order))])

pre = ColumnTransformer([
    ('num', num,  num_cols),
    ('nom', nom,  nom_cols),
    ('ord', ordp, ord_cols)])

model = Pipeline([('pre', pre),
                  ('clf', LogisticRegression())])
model.fit(X_train, y_train)   # fit = train only
joblib.dump(model, 'model.pkl')  # save the WHOLE thing

Pro · Decide

Which Transform? A 60-Second Guide

🧭

When in doubt, StandardScaler

It's the safe default for linear models, SVM and PCA. Switch to MinMax for neural nets, Robust when legitimate outliers must stay — and always fix heavy skew before any scaler touches the column.

Pro · Debug

Common Pitfalls & Their Fixes

Symptom	The trap	The fix
Inflated validation score	Fit scaler on full data	Fit on `X_train` only — use a Pipeline
One feature dominates KNN	No scaling at all	`StandardScaler` / `MinMaxScaler`
Skew survives scaling	Scaling a skewed column	`log1p` / power transform first
Nonsense linear coefficients	Label-encoded nominal cols	`OneHotEncoder`
Thousands of columns	One-hot on high cardinality	Target / frequency encoding
Model fails at inference	Saved model, not the scaler	`joblib.dump(pipeline)`
Scaled values look wrong	Scaled before cleaning	Clean → transform, in that order

Section 10 · Part 1

The Golden Rules — 1 to 4

⚖️ DATA TRANSFORMATION · RULES 1–4

1

Fit on train only. Fitting a scaler on the full dataset leaks test statistics — a subtle leak that inflates every validation score.

2

Pick by algorithm. Standard for linear/PCA/SVM, MinMax for neural nets, Robust for legit extremes — and skip scaling for trees.

3

Check skew before scaling. If |df['col'].skew()| > 1, log or power-transform first — scaling only rescales skew, never removes it.

4

Never label-encode nominals for linear models or nets — it invents an order. One-hot instead.

Section 10 · Part 2

The Golden Rules — 5 to 8 & Takeaway

⚖️ DATA TRANSFORMATION · RULES 5–8

5

Tame high cardinality. For >15 categories, skip one-hot — use target (with CV) or frequency encoding to dodge the curse of dimensionality.

6

Automate with pipelines. Pipeline + ColumnTransformer kill hand-applied, train-inference drift by design.

7

Save the fitted pipeline, not just the model — joblib.dump(pipeline). A model without its scaler is unusable.

8

Verify with distributions. describe() and histograms after every step — and clean before you transform.

🌉

The bridge to learnable patterns

Transformation isn't preprocessing bureaucracy — it's what turns raw numbers into signal a model can learn. The fintech team's 71%→88% jump wasn't a model or data problem. It was a transformation problem, solved in four lines.

⚖️ End of tutorial · Press ← to review, or click Restart