🔡

Encoding Categorical Variables

Turning text labels into numbers a model can learn from — without inventing false relationships. The right encoding is a modelling decision, not a mechanical translation.

Label & Ordinal One-Hot Target & Frequency Pipelines

Press Next → or use ← → arrow keys

Section 01

Why Machines Can't Read Text Labels

🏦 REAL STORY · GINI 0.48 → 0.61

The model that thought Delhi beat Mumbai

A junior data scientist ran LabelEncoder on a city column, so Bengaluru=0 … Pune=4. The logistic-regression loan model duly "learned" that Delhi (2) borrowers were safer than Mumbai (3) — a ranking invented by alphabetical order, not data. Switching to One-Hot lifted the Gini from 0.48 to 0.61 (+27%).

⛔

The cardinal sin of encoding

Label-encoding a nominal (unordered) column for a linear, distance-based or neural model is the most common encoding mistake in data science. It plants a false numeric ordering the model treats as real signal.

Section 01 · Toolkit

Six Ways to Encode a Category

🔢 Label

0,1,2…N

Integers per category. Binary columns & tree models only.

🎫 One-Hot

N binary cols

One 0/1 column per value. Nominal, <15 categories. The gold standard.

📊 Ordinal

ranked ints

Integers that preserve a real order: Low < Med < High.

🎯 Target

mean(target)

Category → mean outcome. High cardinality — but leakage-prone.

📈 Frequency

count / %

Category → how often it appears. Safe, no target used.

🧬 Binary / Hash

log₂(N) cols

Bits instead of columns. Very high cardinality, space-efficient.

🧩

Match the method to the feature

There's no universally "best" encoder. The right choice depends on three things: is the category ordered, how many unique values, and is the model linear or tree-based.

Section 02

Label Encoding — When Is It Safe?

from sklearn.preprocessing import LabelEncoder
le = LabelEncoder()
df['gender_enc'] = le.fit_transform(df['gender'])
dict(zip(le.classes_, le.transform(le.classes_)))   # {'Female':0,'Male':1,'Other':2}
# explicit .map() is always safer — you control the numbers
df['gender_enc'] = df['gender'].map({'Female':0, 'Male':1, 'Other':2})

Section 02 · The Trap

The False-Distance Problem

Text label

gender	returned
Male	Yes
Female	No
Other	Yes

Label-encoded

gender_enc	ret_enc
1	1
0	0
2	1

✅

Binary is genuinely safe

With only two values, "0 vs 1" carries no false ordering — there's nothing to mis-rank. The danger begins at three or more unordered categories.

Section 03

One-Hot Encoding — The Gold Standard

Rebuilding a recommender with 28 cities as 28 independent binary columns — instead of one invented number line — let the model learn a separate weight per city. Click-through rose 34% in A/B testing.

Section 03 · Code

Three Ways to One-Hot & the Dummy Trap

# 1 · quickest for notebooks
pd.get_dummies(df, columns=['city'], dtype=int)

# 2 · drop one col → avoid trap
pd.get_dummies(df, columns=['city'],
               drop_first=True, dtype=int)

# 3 · sklearn, pipeline-friendly
ohe = OneHotEncoder(
    sparse_output=False,
    handle_unknown='ignore',  # unseen → 0s
    drop='first')
ohe.fit(X_train[['city']])
ohe.get_feature_names_out(['city'])

🪤

The dummy-variable trap

N categories need only N−1 columns — if all others are 0, the last is implied. Keeping all N makes the columns perfectly collinear and destabilises linear coefficients.

🌳

Trees don't care

Use drop='first' for linear models. Tree models are immune to collinearity — keep all N.

📏

The 15-category rule

Above ~15 unique values, one-hot explodes the feature space (postcode: 6000+ columns). That's the curse of dimensionality — switch to target or frequency encoding.

Section 04

Ordinal Encoding — When Order Is Real

from sklearn.preprocessing import OrdinalEncoder
oe = OrdinalEncoder(categories=[['Low','Medium','High','Very High']],
                    handle_unknown='use_encoded_value', unknown_value=-1)
df[['income_enc']] = oe.fit_transform(df[['income_bracket']])   # Low→0 … Very High→3

Section 05

Target Encoding — Taming High Cardinality

An e-commerce platform had 8,200 pin codes — one-hot would mean 8,200 dead columns. Replacing each with its mean delivery rating packed the full signal into one column and lifted R² from 0.61 to 0.79.

# category_encoders handles smoothing + CV for you
import category_encoders as ce
te = ce.TargetEncoder(cols=['city'], smoothing=10)
X_train['city_te'] = te.fit_transform(X_train[['city']], y_train)   # fit on TRAIN
X_test['city_te']  = te.transform(X_test[['city']])

Section 05 · Leakage

Target Encoding's Deadly Pitfall

🔓

Naïve target encoding peeks at the answer

Compute category means on the same rows you train on, and the encoded value already contains the target — validation scores soar, production collapses. Always use K-Fold (or a holdout), or let category_encoders smooth it for you.

Section 06

Frequency Encoding — Safe & Simple

# proportion of rows per category
freq = df['city'].value_counts(
           normalize=True)
df['city_freq'] = df['city'].map(freq)

# raw counts variant
df['city_count'] = df['city'].map(
    df['city'].value_counts())

🛟

No target, no leakage

Frequency encoding never touches the label, so it's leak-safe by construction — and the signal is real: rare categories often behave differently from common ones.

Section 07

Which Encoding Wins? It Depends on the Model

📈

Read the two curves

Label encoding cripples logistic regression (false ordering) but barely dents Random Forest. One-hot rescues the linear model; target encoding — done with CV — tends to top both. The encoder and the algorithm are one decision, not two.

Section 07 · Reference

The Encoder Cheat-Sheet

Method	Output cols	High cardinality	Leakage	Best for
Label	1	ok	none	Binary, tree models only
One-Hot	N	>15 bad	none	Nominal, low-card, linear
Ordinal	1	ok	none	Ordered: Low/Med/High
Target	1	yes	high — CV	High-card, strong signal
Frequency	1	yes	none	High-card, safe default
Binary	log₂(N)	yes	none	Very high-card, compact

🧭

The 30-second rule

Ordered? Ordinal. Nominal & small? One-hot. Nominal & huge? Target (with CV) or frequency. Binary or tree model? Label is fine.

Section 08

One Leak-Proof Encoding Pipeline

nom = Pipeline([('imp', SimpleImputer('most_frequent')),
                ('oh', OneHotEncoder(
                        handle_unknown='ignore',
                        drop='first'))])
ordp = Pipeline([('imp', SimpleImputer('most_frequent')),
                 ('ord', OrdinalEncoder(
                          categories=order))])

pre = ColumnTransformer([
    ('num', num_pipe, numeric_cols),
    ('nom', nom,      nominal_cols),
    ('ord', ordp,     ordinal_cols)])

pipe = Pipeline([('pre', pre),
                 ('clf', RandomForestClassifier())])
pipe.fit(X_train, y_train)     # fit = train only
joblib.dump(pipe, 'pipeline.pkl')

Pro · Compact

Binary Encoding — Bits Beat Columns

When a column has thousands of categories, even target encoding may lose nuance. Binary encoding maps each category to an integer, then to its binary bits — so N categories need only log₂(N) columns instead of N.

import category_encoders as ce
be = ce.BinaryEncoder(cols=['product_sku'])
X_train = be.fit_transform(X_train)     # log₂(N) columns, no target used

Pro · Debug

Common Pitfalls & Their Fixes

Symptom	The trap	The fix
Linear model learns nonsense	Label-encoded nominal col	`OneHotEncoder`
Ranking backwards	Alphabetical label on ordered col	`OrdinalEncoder(categories=[…])`
Thousands of columns	One-hot on high cardinality	Target / frequency / binary
Great CV, awful in prod	Target encode on full data	K-Fold CV or `category_encoders`
Unstable coefficients	Kept all N one-hot columns	`drop='first'` (linear only)
Crash on new category	No unknown handling	`handle_unknown='ignore'`
Model unusable at inference	Saved model, not encoders	`joblib.dump(pipeline)`

Section 09 · Part 1

The Golden Rules — 1 to 4

🔡 CATEGORICAL ENCODING · RULES 1–4

1

Never label-encode nominals for linear, SVM, KNN or neural models — it invents an order they read as signal. Binary columns and tree models only.

2

Specify order explicitly with OrdinalEncoder(categories=[…]) — alphabetical is almost never the true rank.

3

Handle unknowns. handle_unknown='ignore' (one-hot) and 'use_encoded_value' (ordinal) — production always brings unseen categories.

4

Drop one column after one-hot for linear models (drop='first') to dodge the dummy-variable trap. Trees keep all N.

Section 09 · Part 2

The Golden Rules — 5 to 8 & Takeaway

🔡 CATEGORICAL ENCODING · RULES 5–8

5

Switch at 15 uniques. Above ~15 categories, drop one-hot for target (with CV) or frequency encoding.

6

Never leak the target. Compute target means with K-Fold or a holdout — never on the rows you train on.

7

Fit on train only, then transform() the test set — fitting on everything leaks category stats.

8

Wrap it in a Pipeline. ColumnTransformer guarantees identical encoding at train and inference — and save the whole thing.

🎯

Encoding is a modelling decision

The encoder decides what relationships the model can even learn from a feature. The team that swapped Label for One-Hot didn't write a better model — they handed it better information. That's the difference between good and exceptional.

🔡 End of tutorial · Press ← to review, or click Restart