Data Transformation — Normalisation & Standardisation in Python

Why Data Transformation Matters 🔧Data Transformation क्यों ज़रूरी है 🔧为什么需要数据变换 🔧

Raw data comes in units that suit people: age in years, salary in rupees, credit score in points. Many algorithms only see numbers. If salary runs into lakhs and dependents run from 0 to 8, a distance-based model like KNN measures almost everything in rupees and ignores the rest. Data transformation puts features on a fair footing — by scaling them, reshaping skewed ones, and turning categories into numbers.

Raw data लोगों के हिसाब की इकाइयों में आता है: उम्र सालों में, salary रुपयों में, credit score points में। कई algorithms सिर्फ़ संख्याएँ देखते हैं। अगर salary लाखों में है और dependents 0 से 8 तक, तो KNN जैसा दूरी-आधारित model लगभग सब कुछ रुपयों में मापता है और बाक़ी को नज़रअंदाज़ कर देता है। Data transformation features को बराबरी पर लाता है — उन्हें scale करके, skewed features का आकार बदलकर, और categories को संख्याओं में बदलकर।

原始数据使用适合人类的单位：年龄用年、薪资用卢比、信用分用分数。许多算法只看数字。如果薪资是几十万、家属人数是 0 到 8，KNN 这类基于距离的模型几乎完全用卢比来衡量，其他特征被忽略。数据变换让各特征处于公平的起点 —— 通过缩放、重塑偏态特征，以及把类别变成数字。

# 1) Why scale? KNN on raw features vs scaled features
import numpy as np, pandas as pd
from sklearn.model_selection import train_test_split
from sklearn.neighbors import KNeighborsClassifier
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import make_pipeline

df = pd.read_csv("loans.csv", parse_dates=["registration_date", "application_date"])
num = ["age", "years_employed", "dependents", "salary", "credit_score", "loan_amount"]
X_train, X_test, y_train, y_test = train_test_split(df[num], df["default"], test_size=0.25,
                                                    random_state=0, stratify=df["default"])
print(df[num].agg(["min", "max"]).T.astype(int))

sq = (X_train - X_train.mean()) ** 2                     # typical squared gap per column
print("\nshare of squared distance from each column (raw):")
print((sq.mean() / sq.mean().sum() * 100).round(3).to_string())

raw = KNeighborsClassifier(n_neighbors=15).fit(X_train, y_train)
scaled = make_pipeline(StandardScaler(), KNeighborsClassifier(n_neighbors=15)).fit(X_train, y_train)
print(f"\nKNN accuracy  raw: {raw.score(X_test, y_test):.3f}   scaled: {scaled.score(X_test, y_test):.3f}"
      f"   (always guessing 'no default': {1 - y_test.mean():.3f})")

The Main Tools at a Glance 🧰मुख्य Tools एक नज़र में 🧰主要工具一览 🧰

Min-Max Normalisation 📏Min-Max Normalisation 📏Min-Max 归一化 📏

Min-Max maps the smallest training value to 0 and the largest to 1. The shape of the distribution does not change — a skewed column stays exactly as skewed. It is popular for neural-network inputs and image pixels (divide by 255).

Min-Max सबसे छोटे training मान को 0 और सबसे बड़े को 1 पर ले जाता है। Distribution का आकार नहीं बदलता — skewed column ठीक उतना ही skewed रहता है। Neural-network inputs और image pixels (255 से भाग) के लिए यह लोकप्रिय है।

Min-Max 把训练集最小值映射为 0，最大值映射为 1。分布形状不变 —— 偏态列的偏度完全保持不变。它常用于神经网络输入和图像像素（除以 255）。

# 2) Min-Max: fit on TRAIN, then transform both
from sklearn.preprocessing import MinMaxScaler
mm = MinMaxScaler()                                      # feature_range=(0, 1) by default
Xtr_mm = pd.DataFrame(mm.fit_transform(X_train), columns=num, index=X_train.index)
Xte_mm = pd.DataFrame(mm.transform(X_test), columns=num, index=X_test.index)
print("train range:", Xtr_mm.min().min().round(3), "to", Xtr_mm.max().max().round(3))
print("test  range:", Xte_mm.min().min().round(3), "to", Xte_mm.max().max().round(3), "← new data CAN fall outside [0, 1]; that is expected")
print("salary skew before", X_train["salary"].skew().round(2), "after", Xtr_mm["salary"].skew().round(2), "(shape unchanged)")
back = mm.inverse_transform(Xtr_mm)[:2, 3]
print("inverse_transform gives back salaries:", back.round(0), "| original:", X_train["salary"].iloc[:2].to_numpy())

Standardisation (Z-Score) 📐Standardisation (Z-Score) 📐标准化（z 分数）📐

Standardising subtracts the mean and divides by the standard deviation, so each feature has mean 0 and std 1. It does not cap values and does not require normal data. It is the usual default for linear and logistic regression, PCA and SVM.

Standardising, mean घटाता है और standard deviation से भाग देता है, ताकि हर feature का mean 0 और std 1 हो। यह मानों पर सीमा नहीं लगाता और normal data की ज़रूरत नहीं रखता। Linear और logistic regression, PCA और SVM के लिए यह आम default है।

标准化减去均值再除以标准差，使每个特征均值为 0、标准差为 1。它不限制取值范围，也不要求数据服从正态分布。它是线性回归、逻辑回归、PCA 和 SVM 的常用默认做法。

# 3) Standardisation: mean 0, std 1 — and comparable coefficients
from sklearn.linear_model import LogisticRegression
sc = StandardScaler().fit(X_train)
Z = pd.DataFrame(sc.transform(X_train), columns=num)
print(Z.agg(["mean", "std"]).round(3))
print("note: sklearn divides by the population std (ddof=0), pandas .std() uses ddof=1 → tiny difference")
lr_std = LogisticRegression(max_iter=1000).fit(Z, y_train)
coef = pd.Series(lr_std.coef_[0], index=num).sort_values(key=abs, ascending=False)
print("\nstandardised coefficients (effect of +1 std):"); print(coef.round(3).to_string())

Robust Scaling — When Extremes Are Real 🛡️Robust Scaling — जब अति मान असली हों 🛡️稳健缩放 —— 当极端值真实存在时 🛡️

If a few very large values are genuine (a ₹6 crore salary does exist), you may want to keep them. RobustScaler centres on the median and divides by the IQR, so those few values cannot squash everyone else.

अगर कुछ बहुत बड़े मान असली हैं (₹6 करोड़ की salary होती है), तो आप उन्हें रखना चाह सकते हैं। RobustScaler median पर केंद्रित करता है और IQR से भाग देता है, इसलिए वे कुछ मान बाक़ी सबको दबा नहीं सकते।

如果少数极大值是真实的（确实存在 ₹6 千万的年薪），你可能想保留它们。RobustScaler 以中位数为中心、除以 IQR，所以这几个值无法把其他所有人挤压在一起。

# 4) One extreme value: MinMax vs Standard vs Robust vs MaxAbs
from sklearn.preprocessing import RobustScaler, MaxAbsScaler
s = X_train[["salary"]].copy()
s.iloc[0, 0] = 60_000_000                                # one real but extreme salary: ₹6 crore
mid = s["salary"].between(*s["salary"].quantile([0.25, 0.75]))
for name, scaler in [("MinMax", MinMaxScaler()), ("Standard", StandardScaler()), ("Robust", RobustScaler()), ("MaxAbs", MaxAbsScaler())]:
    v = scaler.fit_transform(s).ravel()
    print(f"{name:<8} middle 50% spans {v[mid].min():7.3f} to {v[mid].max():6.3f}  (width {v[mid].max() - v[mid].min():.3f})   extreme → {v.max():7.2f}")

Log and Power Transforms — Changing the Shape 📉Log और Power Transforms — आकार बदलना 📉对数与幂变换 —— 改变形状 📉

Scalers change the range; they keep the shape. A long right tail — common in loan amounts, incomes and page views — needs a shape-changing transform. These help linear models, which can be dominated by a few huge values, and they often make relationships more linear.

Scalers range बदलते हैं; आकार बना रहता है। लंबी right tail — जो loan amounts, incomes और page views में आम है — के लिए आकार बदलने वाला transform चाहिए। ये linear models की मदद करते हैं, जिन पर कुछ बहुत बड़े मान हावी हो सकते हैं, और अक्सर संबंधों को ज़्यादा linear बनाते हैं।

缩放器改变的是范围，保留的是形状。贷款金额、收入、页面浏览量中常见的右侧长尾，需要能改变形状的变换。这对可能被少数巨大值主导的线性模型很有帮助，也常能让关系更接近线性。

# 5) Changing the SHAPE: log, sqrt, Yeo-Johnson, Box-Cox, quantile
from sklearn.preprocessing import PowerTransformer, QuantileTransformer
x = X_train[["loan_amount"]]
out = {"original": x["loan_amount"],
       "sqrt": np.sqrt(x["loan_amount"]),
       "log1p": np.log1p(x["loan_amount"])}
yj = PowerTransformer(method="yeo-johnson").fit(x)
bc = PowerTransformer(method="box-cox").fit(x)                         # needs x > 0
qt = QuantileTransformer(output_distribution="normal", n_quantiles=1000, random_state=42).fit(x)
out["yeo-johnson"] = pd.Series(yj.transform(x).ravel())
out["box-cox"] = pd.Series(bc.transform(x).ravel())
out["quantile→normal"] = pd.Series(qt.transform(x).ravel())
for k, v in out.items():
    print(f"{k:<16} skew {v.skew():6.2f}")
print(f"lambda: Yeo-Johnson {yj.lambdas_[0]:.3f}, Box-Cox {bc.lambdas_[0]:.3f}  (0 would be a pure log)")

Encoding Categories 🏷️Categories को Encode करना 🏷️类别编码 🏷️

Most models need numbers, so text columns must be encoded. The right method depends on whether the categories have a real order, how many there are, and whether you may use the target.

ज़्यादातर models को संख्याएँ चाहिए, इसलिए text columns को encode करना पड़ता है। सही तरीका इस पर निर्भर है कि categories में असली क्रम है या नहीं, कितनी हैं, और क्या आप target इस्तेमाल कर सकते हैं।

大多数模型需要数字，所以文本列必须编码。正确的方法取决于类别是否有真实顺序、类别有多少，以及能否使用目标变量。

# 6) Encoding categories
from sklearn.preprocessing import OneHotEncoder, OrdinalEncoder, TargetEncoder
from sklearn.metrics import roc_auc_score
from sklearn.model_selection import KFold
tr, te = train_test_split(df, test_size=0.25, random_state=0, stratify=df["default"])

ohe = OneHotEncoder(handle_unknown="ignore", sparse_output=False).fit(tr[["city"]])
print("one-hot columns:", list(ohe.get_feature_names_out()))
oe = OrdinalEncoder(categories=[["Low", "Medium", "High", "Very High"]]).fit(tr[["income_bracket"]])
print("ordinal:", dict(zip(oe.categories_[0], range(4))))

# branch_id has 400 values and NO real link to default. Naive target encoding still "finds" one:
means = tr.groupby("branch_id")["default"].mean()
naive_tr, naive_te = tr["branch_id"].map(means), te["branch_id"].map(means).fillna(tr["default"].mean())
print(f"\nnaive target encoding  AUC train {roc_auc_score(tr['default'], naive_tr):.3f}   test {roc_auc_score(te['default'], naive_te):.3f}")
tenc = TargetEncoder(cv=KFold(5, shuffle=True, random_state=0))   # cross-fitting inside fit_transform
cf_tr = tenc.fit_transform(tr[["branch_id"]].astype(str), tr["default"]).ravel()
cf_te = tenc.transform(te[["branch_id"]].astype(str)).ravel()
print(f"TargetEncoder (cv)     AUC train {roc_auc_score(tr['default'], cf_tr):.3f}   test {roc_auc_score(te['default'], cf_te):.3f}")
freq = tr["branch_id"].map(tr["branch_id"].value_counts(normalize=True))
print("frequency encoding example:", freq.head(3).round(4).tolist())

Feature Engineering — Creating New Signals 💡Feature Engineering — नए संकेत बनाना 💡特征工程 —— 创造新信号 💡

Sometimes the useful signal is a combination of columns. In lending, what matters is not the loan or the salary alone but the loan-to-salary ratio. A linear model cannot build that ratio by itself, so we give it one.

कभी-कभी काम का संकेत columns का मेल होता है। Lending में अकेला loan या अकेली salary नहीं, बल्कि loan-to-salary ratio मायने रखता है। Linear model यह ratio ख़ुद नहीं बना सकता, इसलिए हम उसे देते हैं।

有时有用的信号是多个列的组合。在放贷中，重要的不是贷款额或薪资本身，而是贷款与薪资之比。线性模型自己构造不出这个比值，所以我们替它构造。

# 7) Feature engineering: a ratio can matter more than its parts
from sklearn.model_selection import cross_val_score
d = df.copy()
d["loan_to_salary"] = d["loan_amount"] / d["salary"]
d["log_loan_to_salary"] = np.log(d["loan_to_salary"])
snapshot = pd.Timestamp("2024-12-31")                     # fixed date, NOT today(): reproducible
d["days_since_reg"] = (snapshot - d["registration_date"]).dt.days
d["days_reg_to_app"] = (d["application_date"] - d["registration_date"]).dt.days
d["app_month"] = d["application_date"].dt.month
d["age_group"] = pd.cut(d["age"], bins=[20, 30, 40, 50, 66], labels=["21-30", "31-40", "41-50", "51-65"])

model = make_pipeline(StandardScaler(), LogisticRegression(max_iter=1000))
for cols in [num, num + ["log_loan_to_salary"]]:
    auc = cross_val_score(model, d[cols], d["default"], cv=5, scoring="roc_auc").mean()
    print(f"{len(cols)} features {'+ log(loan/salary)' if len(cols) > len(num) else '(raw only)'}".ljust(30) + f"CV AUC {auc:.3f}")
from sklearn.preprocessing import PolynomialFeatures
pf = PolynomialFeatures(degree=2, include_bias=False).fit(d[["age", "dependents"]])
print("polynomial names:", list(pf.get_feature_names_out()))

One Pipeline for Everything 🔗हर चीज़ के लिए एक Pipeline 🔗一条管道搞定一切 🔗

ColumnTransformer sends each group of columns through its own steps; Pipeline chains those steps with the model. Calling fit learns every median, mean, std and category from the training rows only, and predict applies exactly the same steps to new data.

ColumnTransformer columns के हर समूह को उसके अपने क़दमों से गुज़ारता है; Pipeline उन क़दमों को model से जोड़ता है। fit बुलाने पर हर median, mean, std और category सिर्फ़ training rows से सीखी जाती है, और predict नए data पर ठीक वही क़दम लगाता है।

ColumnTransformer 让每组列各自经过对应的步骤；Pipeline 把这些步骤和模型串起来。调用 fit 时，所有中位数、均值、标准差和类别都只从训练行学习；predict 会对新数据执行完全相同的步骤。

# 8) Everything in one Pipeline: no leakage, one object to save
import joblib
from sklearn.compose import ColumnTransformer
from sklearn.impute import SimpleImputer
from sklearn.pipeline import Pipeline
d["log_loan_amount"], d["log_salary"] = np.log(d["loan_amount"]), np.log(d["salary"])
num_cols = ["age", "years_employed", "dependents", "credit_score", "log_salary", "log_loan_amount", "log_loan_to_salary"]
pre = ColumnTransformer([
    ("num", Pipeline([("impute", SimpleImputer(strategy="median")), ("scale", StandardScaler())]), num_cols),
    ("nom", Pipeline([("impute", SimpleImputer(strategy="most_frequent")),
                      ("ohe", OneHotEncoder(handle_unknown="ignore"))]), ["city"]),
    ("ord", OrdinalEncoder(categories=[["Low", "Medium", "High", "Very High"]]), ["income_bracket"]),
])
pipe = Pipeline([("prep", pre), ("model", LogisticRegression(max_iter=1000))])
cols = num_cols + ["city", "income_bracket"]
tr, te = train_test_split(d, test_size=0.25, random_state=0, stratify=d["default"])
pipe.fit(tr[cols], tr["default"])
print(f"test AUC: {roc_auc_score(te['default'], pipe.predict_proba(te[cols])[:, 1]):.3f}")
print("features after ColumnTransformer:", pipe["prep"].get_feature_names_out().size)
joblib.dump(pipe, "loan_pipeline.joblib")
same = (joblib.load("loan_pipeline.joblib").predict_proba(te[cols]) == pipe.predict_proba(te[cols])).all()
print("reloaded pipeline gives identical predictions:", bool(same))

Make the Practice Data 🐍Practice Data बनाएँ 🐍生成练习数据 🐍

Run this first. Default risk is built from credit score, the loan-to-salary ratio, dependents and years employed (plus a little from city). branch_id has 400 values and no effect at all — ideal for showing leakage.

इसे पहले चलाएँ। Default का जोखिम credit score, loan-to-salary ratio, dependents और years employed (और city से थोड़ा) से बना है। branch_id के 400 मान हैं और कोई असर नहीं — leakage दिखाने के लिए आदर्श।

请先运行它。违约风险由信用分、贷款与薪资之比、家属人数和工作年限（外加少量城市因素）决定。branch_id 有 400 个取值，但完全没有影响 —— 非常适合演示泄漏。

# 0) A practice loan dataset — the default risk is planted, so we can check every step
import numpy as np, pandas as pd
rng = np.random.default_rng(5)
n = 4000
age = rng.integers(21, 66, n)
years_employed = np.clip(np.round((age - 21) * rng.uniform(0.2, 0.9, n)), 0, 40).astype(int)
dependents = rng.poisson(1.4, n).clip(0, 8)
salary = np.round(np.exp(rng.normal(13.4, 0.55, n) + 0.01 * years_employed), -3)          # ₹ per year
credit_score = np.clip(np.round(rng.normal(690, 70, n) + 1.2 * years_employed), 300, 900).astype(int)
loan_amount = np.round(salary * rng.uniform(0.2, 2.5, n), -3)
city = rng.choice(["Mumbai", "Delhi", "Bengaluru", "Chennai", "Hyderabad", "Pune", "Kolkata", "Jaipur"], n)
branch_id = rng.integers(1, 401, n)                                                       # 400 branches, no real effect
income_bracket = pd.cut(salary, [0, 4e5, 8e5, 15e5, np.inf], labels=["Low", "Medium", "High", "Very High"]).astype(str)
reg = pd.Timestamp("2018-01-01") + pd.to_timedelta(rng.integers(0, 2000, n), unit="D")
app = reg + pd.to_timedelta(rng.integers(30, 900, n), unit="D")
# risk: low credit score, high loan-to-salary ratio, many dependents, few years employed
z = (-0.012 * (credit_score - 690) + 1.6 * np.log(loan_amount / salary) + 0.35 * dependents
     - 0.06 * years_employed + 0.25 * (city == "Kolkata") - 1.4 + rng.normal(0, 0.5, n))
default = (rng.random(n) < 1 / (1 + np.exp(-z))).astype(int)
pd.DataFrame({"age": age, "years_employed": years_employed, "dependents": dependents, "salary": salary,
              "credit_score": credit_score, "loan_amount": loan_amount, "city": city, "branch_id": branch_id,
              "income_bracket": income_bracket, "registration_date": reg, "application_date": app,
              "default": default}).to_csv("loans.csv", index=False)
print(f"loans.csv: {n:,} rows, default rate {default.mean():.1%}")

Golden Rules of Data Transformation 🏆Data Transformation के सुनहरे नियम 🏆数据变换黄金法则 🏆