Feature Engineering & Feature Scaling

Why Feature Engineering Matters 🛠️Feature Engineering क्यों ज़रूरी है 🛠️为什么特征工程很重要 🛠️

A model can only learn from the columns you give it. If the useful signal is hidden — in the ratio of two columns, inside a timestamp, or in a customer's past behaviour — many models will not find it on their own. Feature engineering builds new columns that put the signal in plain view: ratios, powers, date parts, bins and group averages.

Model सिर्फ़ उन्हीं columns से सीख सकता है जो आप उसे देते हैं। अगर काम का संकेत छिपा है — दो columns के ratio में, timestamp के अंदर, या ग्राहक के पिछले व्यवहार में — तो कई models उसे ख़ुद नहीं ढूँढेंगे। Feature engineering नए columns बनाता है जो संकेत को सामने रख दें: ratios, powers, date के हिस्से, bins और group averages।

模型只能从你提供的列中学习。如果有用的信号藏在别处 —— 两列的比值里、时间戳内部，或客户过去的行为中 —— 很多模型自己找不到它。特征工程就是构造新列，把信号摆到明面上：比率、幂次、日期成分、分箱和分组平均。

Six Kinds of Engineered Features 🧰छह तरह के Engineered Features 🧰六类构造特征 🧰

Ratios and Interactions ✖️Ratios और Interactions ✖️比率与交互 ✖️

Some signals live in the relationship between two columns. A person earning ₹30,000 a month with ₹5 lakh of debt is in a very different position from one earning ₹3 lakh with the same debt. Neither income nor debt says that alone; debt ÷ yearly income does.

कुछ संकेत दो columns के रिश्ते में रहते हैं। ₹30,000 महीना कमाने वाले पर ₹5 लाख का कर्ज़ उस व्यक्ति से बहुत अलग स्थिति है जो उतने ही कर्ज़ के साथ ₹3 लाख कमाता है। यह न अकेले income बताती है न debt; debt ÷ सालाना income बताता है।

有些信号存在于两列的关系中。月入 ₹3 万、负债 ₹50 万的人，与月入 ₹30 万、负债相同的人处境截然不同。单看收入或负债都说明不了这一点；负债 ÷ 年收入可以。

# 1) A ratio feature: debt ÷ yearly income
import numpy as np, pandas as pd
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import make_pipeline
from sklearn.linear_model import LogisticRegression
from sklearn.ensemble import HistGradientBoostingClassifier
from sklearn.metrics import roc_auc_score
loans = pd.read_csv("loans.csv")
loans["debt_to_income"] = loans["total_debt"] / (12 * loans["monthly_income"])   # income is never 0 here
train, test = train_test_split(loans, test_size=0.25, random_state=0, stratify=loans["default"])
y_tr, y_te = train["default"], test["default"]
def auc(model, cols):
    model.fit(train[cols], y_tr)
    return roc_auc_score(y_te, model.predict_proba(test[cols])[:, 1])
logit = make_pipeline(StandardScaler(), LogisticRegression())
for cols in [["monthly_income"], ["total_debt"], ["monthly_income", "total_debt"], ["debt_to_income"]]:
    print(f"logistic on {str(cols):<36} test AUC {auc(logit, cols):.3f}")

Two raw columns together reach 0.691 because a logistic model can partly combine them; the single ratio reaches 0.700 and is far easier to explain. Ratios, products (price × quantity) and differences (balance_end − balance_start) are all built the same way — from the raw values.

दो raw columns मिलकर 0.691 तक पहुँचते हैं क्योंकि logistic model उन्हें कुछ हद तक जोड़ लेता है; अकेला ratio 0.700 तक पहुँचता है और समझाना कहीं आसान है। Ratios, products (price × quantity) और अंतर (balance_end − balance_start) सब एक ही तरह बनते हैं — raw मानों से।

两个原始列一起达到 0.691，因为逻辑模型能部分组合它们；单个比率达到 0.700，而且更容易解释。比率、乘积（price × quantity）和差值（balance_end − balance_start）都以同样方式构造 —— 用原始值。

Dividing safelyसुरक्षित भाग安全地做除法

The original page suggested a / (b + 1) and np.where(b > 0, a / b, 0). Both have a catch. Adding 1 badly distorts small denominators: 900 ÷ 1 day becomes 450. And np.where computes a / b for every row first, so it still divides by zero (with warnings); filling with 0 also pretends "no data" means "zero". Prefer leaving the ratio missing when the denominator is 0 and let the imputer or model handle it.

मूल page ने a / (b + 1) और np.where(b > 0, a / b, 0) सुझाए थे। दोनों में दिक़्क़त है। 1 जोड़ने से छोटे denominators बहुत बिगड़ते हैं: 900 ÷ 1 दिन 450 बन जाता है। और np.where पहले हर row के लिए a / b निकालता है, इसलिए फिर भी शून्य से भाग होता है (warnings के साथ); 0 भरना यह भी दिखावा करता है कि "data नहीं" का मतलब "शून्य" है। Denominator 0 हो तो ratio को missing छोड़ें और imputer या model को संभालने दें।

原页面建议 a / (b + 1) 和 np.where(b > 0, a / b, 0)，两者都有问题。加 1 会严重扭曲小分母：900 ÷ 1 天变成 450。而 np.where 会先对每一行计算 a / b，所以照样除以零（并产生警告）；用 0 填充还假装“没有数据”等于“零”。分母为 0 时，最好让比率保持缺失，交给填补器或模型处理。

# 2) Safe division: "+ 1" changes small denominators a lot; np.where still divides by zero first
pd.set_option("display.width", 160); pd.set_option("display.max_columns", 10)
acct = pd.DataFrame({"spend": [900, 900, 900, 900], "active_days": [0, 1, 3, 30]})
acct["spend_per_day_plus1"] = acct["spend"] / (acct["active_days"] + 1)
with np.errstate(divide="ignore", invalid="ignore"):                  # np.where evaluates BOTH branches
    acct["spend_per_day_where"] = np.where(acct["active_days"] > 0,
                                                     acct["spend"] / acct["active_days"], np.nan)
acct["spend_per_day_safe"] = acct["spend"].div(acct["active_days"].where(lambda s: s > 0))
print(acct.round(1))

Polynomial Features 📈Polynomial Features 📈多项式特征 📈

A linear model draws straight lines. In our loans, default is high for young borrowers, lowest around 45 and high again after 60 — a U-shape. Adding age² lets the same linear model draw that U. PolynomialFeatures adds every square and cross-product at once; with many columns that grows fast (degree 2 on 20 columns gives 230 columns).

Linear model सीधी रेखाएँ बनाता है। हमारे loans में default युवा borrowers में ज़्यादा है, लगभग 45 पर सबसे कम और 60 के बाद फिर ज़्यादा — U-आकार। age² जोड़ने से वही linear model U बना सकता है। PolynomialFeatures हर square और cross-product एक साथ जोड़ता है; कई columns पर यह तेज़ी से बढ़ता है (20 columns पर degree 2 से 230 columns)।

线性模型只能画直线。在我们的贷款数据中，年轻借款人违约率高，45 岁左右最低，60 岁以后又升高 —— 呈 U 形。加入 age² 后，同一个线性模型就能画出这个 U。PolynomialFeatures 会一次性加入所有平方项和交叉项；列多时增长很快（20 列做 2 次展开得到 230 列）。

# 3) Polynomial features: a U-shaped age effect, and why centring matters
from sklearn.preprocessing import PolynomialFeatures
base = ["age", "credit_score", "years_employed", "debt_to_income"]
print(f"logistic, age as a straight line : test AUC {auc(logit, base):.3f}")
poly_age = make_pipeline(StandardScaler(), PolynomialFeatures(degree=2, include_bias=False), StandardScaler(), LogisticRegression())
print(f"logistic, all degree-2 terms     : test AUC {auc(poly_age, base):.3f}")
train = train.assign(age_sq=train["age"] ** 2); test = test.assign(age_sq=test["age"] ** 2)
print(f"logistic, + age² only            : test AUC {auc(logit, base + ['age_sq']):.3f}")
pf = PolynomialFeatures(degree=2, include_bias=False).fit(train[["age", "credit_score"]])
print("\nnames:", pf.get_feature_names_out().tolist())
age_c = train["age"] - train["age"].mean()
print(f"corr(age, age²) = {np.corrcoef(train['age'], train['age'] ** 2)[0, 1]:.3f} raw  vs  "
      f"{np.corrcoef(age_c, age_c ** 2)[0, 1]:.3f} after centring")

Date and Time Features 📅Date और Time Features 📅日期与时间特征 📅

A timestamp hides many signals: hour of day, day of week, month, holidays, and how long ago something happened. As one big number (seconds since 1970) it only says "earlier or later" — a linear model can draw just one slope through it, and a tree must cut it into thousands of tiny ranges to find a weekly pattern.

Timestamp में कई संकेत छिपे होते हैं: दिन का hour, हफ़्ते का दिन, महीना, छुट्टियाँ, और कोई चीज़ कितने समय पहले हुई। एक बड़ी संख्या (1970 से seconds) के रूप में यह बस "पहले या बाद" बताता है — linear model उस पर एक ही ढलान बना सकता है, और tree को weekly pattern ढूँढने के लिए उसे हज़ारों छोटे हिस्सों में काटना पड़ता है।

时间戳里藏着很多信号：一天中的小时、星期几、月份、节假日，以及事情发生在多久以前。如果只当作一个大数字（1970 年以来的秒数），它只能表示“更早或更晚” —— 线性模型只能画一条斜线，树模型则要切成成千上万个小区间才能找到每周规律。

# 4) Date and time features for delivery time (train on Jan–Sep, test on Oct–Dec)
from sklearn.linear_model import LinearRegression
from sklearn.ensemble import HistGradientBoostingRegressor
from sklearn.metrics import root_mean_squared_error
orders = pd.read_csv("orders.csv", parse_dates=["order_time"])
t = orders["order_time"]
orders["unix_time"] = t.astype("int64") // 10**9                       # seconds since 1970
orders["hour"] = t.dt.hour
orders["day_of_week"] = t.dt.dayofweek                                  # 0 = Monday
orders["is_friday"] = (orders["day_of_week"] == 4).astype(int)
orders["is_weekend"] = (orders["day_of_week"] >= 5).astype(int)
orders["hour_sin"] = np.sin(2 * np.pi * orders["hour"] / 24)
orders["hour_cos"] = np.cos(2 * np.pi * orders["hour"] / 24)
o_tr, o_te = orders[t < "2025-10-01"], orders[t >= "2025-10-01"]
sets = {"distance + unix_time": ["distance_km", "unix_time"],
        "+ hour, day_of_week (raw numbers)": ["distance_km", "hour", "day_of_week"],
        "+ is_friday, is_weekend": ["distance_km", "hour", "is_friday", "is_weekend"],
        "+ hour_sin, hour_cos (no raw hour)": ["distance_km", "hour_sin", "hour_cos", "is_friday", "is_weekend"]}
print(f"{'features':<38}{'linear RMSE':>12}{'boosting RMSE':>15}")
for name, cols in sets.items():
    r = [root_mean_squared_error(o_te["delivery_hours"], m.fit(o_tr[cols], o_tr["delivery_hours"]).predict(o_te[cols]))
         for m in (LinearRegression(), HistGradientBoostingRegressor(random_state=0))]
    print(f"{name:<38}{r[0]:>12.2f}{r[1]:>15.2f}")
print(f"\n(always guessing the training mean: RMSE {root_mean_squared_error(o_te['delivery_hours'], np.full(len(o_te), o_tr['delivery_hours'].mean())):.2f})")

Binning — Grouping Numbers 📦Binning — संख्याओं को Group करना 📦分箱 —— 把数值分组 📦

Binning turns a number into groups: age 21–30, 31–40 and so on. One-hot bins let a linear model learn a different level for each group — a simple stair-step instead of a smooth curve. pd.cut makes bins of equal width; pd.qcut makes bins with equal counts. Binning throws away detail inside each bin, so use it when the steps are real (tax slabs, age bands) or the model needs it, not by default.

Binning संख्या को groups में बदलता है: age 21–30, 31–40 वगैरह। One-hot bins linear model को हर group के लिए अलग स्तर सीखने देते हैं — smooth curve की जगह सरल सीढ़ी। pd.cut बराबर चौड़ाई के bins बनाता है; pd.qcut बराबर गिनती वाले। Binning हर bin के अंदर का detail फेंक देता है, इसलिए इसे तब इस्तेमाल करें जब steps असली हों (tax slabs, age bands) या model को चाहिए, default के रूप में नहीं।

分箱把数值变成组：年龄 21–30、31–40 等。独热编码后的分箱让线性模型为每组学习不同水平 —— 用简单台阶代替平滑曲线。pd.cut 生成宽度相等的箱；pd.qcut 生成数量相等的箱。分箱会丢掉箱内细节，所以只在台阶真实存在（税档、年龄段）或模型确实需要时使用，而不是默认使用。

# 5) Binning: equal width (cut) vs equal count (qcut) on a skewed column, and bins as a cheap curve
from sklearn.preprocessing import KBinsDiscretizer
from sklearn.compose import ColumnTransformer
print("pd.cut, 5 equal-width bins :", pd.cut(train["monthly_income"], bins=5).value_counts(sort=False).tolist())
print("pd.qcut, 5 equal-count bins:", pd.qcut(train["monthly_income"], q=5).value_counts(sort=False).tolist())
age_groups = pd.cut(train["age"], bins=[20, 30, 40, 50, 60, 70], labels=["21-30", "31-40", "41-50", "51-60", "61-70"])
print("\ndefault rate by age group:", train.groupby(age_groups, observed=True)["default"].mean().round(3).to_dict())
binned = make_pipeline(
    ColumnTransformer([("age_bins", KBinsDiscretizer(n_bins=5, encode="onehot-dense", strategy="quantile", quantile_method="averaged_inverted_cdf"), ["age"])],
                      remainder=StandardScaler()),
    LogisticRegression())
print(f"\nlogistic, age in 5 bins (fitted on train): test AUC {auc(binned, base):.3f}")

Group Aggregates — Only Use the Past 📊Group Aggregates — सिर्फ़ अतीत इस्तेमाल करें 📊分组聚合 —— 只用过去的数据 📊

A single order says little; a customer's history says a lot. Group features such as "this customer's average delivery time" or "this product's return rate" are often among the strongest columns. But they are easy to leak: if the average includes the current row, or orders that happen later, the model is partly reading the answer.

एक अकेला order कम बताता है; ग्राहक का इतिहास बहुत कुछ। "इस ग्राहक का औसत delivery समय" या "इस product की return दर" जैसे group features अक्सर सबसे मज़बूत columns में होते हैं। पर इनमें leak आसान है: अगर औसत में मौजूदा row या बाद में होने वाले orders शामिल हों, तो model कुछ हद तक जवाब पढ़ रहा है।

单个订单说明不了多少；客户的历史却能说明很多。“该客户的平均配送时间”或“该商品的退货率”这类分组特征，往往是最强的列之一。但它们很容易泄漏：如果平均值包含当前行或之后才发生的订单，模型就在部分偷看答案。

# 6) Group features: a customer's usual delivery time — three ways to compute it
best = ["distance_km", "hour_sin", "hour_cos", "is_friday", "is_weekend"]
orders = orders.sort_values("order_time")
g = orders.groupby("customer_id")["delivery_hours"]
orders["cust_avg_ALL"] = g.transform("mean")                                     # wrong: uses every order, even future ones and itself
orders["cust_avg_PAST"] = g.transform(lambda s: s.shift(1).expanding().mean())   # right: only orders BEFORE this one
orders["cust_avg_PAST"] = orders["cust_avg_PAST"].fillna(o_tr["delivery_hours"].mean())   # first order: use the overall mean
o_tr, o_te = orders[orders["order_time"] < "2025-10-01"], orders[orders["order_time"] >= "2025-10-01"]
for name, extra in [("no customer feature", []), ("cust_avg_ALL  (leaks)", ["cust_avg_ALL"]), ("cust_avg_PAST (honest)", ["cust_avg_PAST"])]:
    lin = LinearRegression().fit(o_tr[best + extra], o_tr["delivery_hours"])
    print(f"{name:<24} test RMSE {root_mean_squared_error(o_te['delivery_hours'], lin.predict(o_te[best + extra])):.2f}")

Domain Features 🧠Domain Features 🧠领域特征 🧠

The best features often come from asking an expert: "What do you calculate in your head when you judge a case?" Their answers are usually ratios, differences and flags.

सबसे अच्छे features अक्सर किसी expert से पूछने से आते हैं: "किसी case को परखते समय आप मन में क्या गिनते हैं?" उनके जवाब आमतौर पर ratios, अंतर और flags होते हैं।

最好的特征往往来自请教专家：“你判断一个案例时，心里会算些什么？”答案通常是比率、差值和标记。

Scaling Comes After Building Ratios ⚖️Scaling Ratios बनाने के बाद ⚖️先构造比率，再缩放 ⚖️

Scaling makes columns comparable for distance- and gradient-based models (see the Feature Scaling tutorial for the four scalers). The order matters: a standardised column is centred at 0, so dividing by it means dividing by numbers near zero, and both signs mix. We measured it.

Scaling दूरी- और gradient-आधारित models के लिए columns को तुलनीय बनाता है (चार scalers के लिए Feature Scaling tutorial देखें)। क्रम मायने रखता है: standardised column 0 पर centred है, इसलिए उससे भाग देना शून्य के पास की संख्याओं से भाग देना है, और दोनों signs मिल जाते हैं। हमने मापा।

缩放让列在基于距离和梯度的模型中可比较（四种缩放器见“特征缩放”教程）。顺序很重要：标准化后的列以 0 为中心，用它做除数就等于除以接近零的数，而且正负号混杂。我们做了测量。

# 7) Order matters: build the ratio from RAW values, then scale
sc = StandardScaler().fit(train[["total_debt", "monthly_income"]])
z_tr = pd.DataFrame(sc.transform(train[["total_debt", "monthly_income"]]), columns=["debt_z", "income_z"], index=train.index)
z_te = pd.DataFrame(sc.transform(test[["total_debt", "monthly_income"]]), columns=["debt_z", "income_z"], index=test.index)
train = train.assign(ratio_of_scaled=z_tr["debt_z"] / z_tr["income_z"]); test = test.assign(ratio_of_scaled=z_te["debt_z"] / z_te["income_z"])
print("ratio of scaled values, min / median / max:", train["ratio_of_scaled"].quantile([0, .5, 1]).round(1).tolist())
for col in ["debt_to_income", "ratio_of_scaled"]:
    print(f"{col:<16} alone → test AUC {auc(logit, [col]):.3f}")

The ratio of scaled values swings from −2,097 to +1,213 and is almost useless (AUC 0.514), while the same ratio from raw values scores 0.700. So: clean → build features from raw values → encode → scale (fit on train) → select → model, all in one pipeline. The scaler then handles the new columns too — engineered features do not arrive pre-scaled.

Scaled मानों का ratio −2,097 से +1,213 तक झूलता है और लगभग बेकार है (AUC 0.514), जबकि raw मानों से वही ratio 0.700 देता है। इसलिए: साफ़ करें → raw मानों से features बनाएँ → encode → scale (train पर fit) → select → model, सब एक pipeline में। तब scaler नए columns को भी संभालता है — engineered features पहले से scaled नहीं आते।

用缩放后的值算出的比率在 −2,097 到 +1,213 之间剧烈波动，几乎无用（AUC 0.514），而用原始值算出的同一比率达到 0.700。所以：清洗 → 用原始值构造特征 → 编码 → 缩放（在训练集拟合）→ 选择 → 模型，全部放进一个管道。缩放器也会处理新列 —— 构造出的特征不会自带缩放。

One Pipeline for Everything 🔗सब कुछ एक Pipeline में 🔗一个管道搞定一切 🔗

Wrap the engineering in a small transformer class so it runs identically on training data, test data and new applications. Anything it learns — here the mean age used for centring — is learned in fit, from training rows only. Bugs fixed from the original pipeline: .score() returns accuracy, not AUC; Timestamp.today() made features change daily; columns created by the engineer (is_weekend, day_of_week) were silently dropped by the ColumnTransformer; utilisation_pct was never created; and gradient boosting does not need the scalers in front of it.

Engineering को एक छोटी transformer class में रखें ताकि वह training data, test data और नई applications पर एक जैसा चले। जो कुछ वह सीखता है — यहाँ centring के लिए औसत age — वह fit में, सिर्फ़ training rows से सीखा जाता है। मूल pipeline के bugs ठीक किए: .score() accuracy देता है, AUC नहीं; Timestamp.today() से features रोज़ बदलते थे; engineer के बनाए columns (is_weekend, day_of_week) को ColumnTransformer चुपचाप गिरा देता था; utilisation_pct कभी बना ही नहीं; और gradient boosting को अपने आगे scalers की ज़रूरत नहीं।

把特征构造封装成一个小的转换器类，使它在训练数据、测试数据和新申请上运行完全一致。它学习的任何东西 —— 这里是用于中心化的平均年龄 —— 都在 fit 中只从训练行学习。修正了原管道的错误：.score() 返回的是准确率而不是 AUC；Timestamp.today() 让特征每天都变；构造器生成的列（is_weekend、day_of_week）被 ColumnTransformer 悄悄丢弃；utilisation_pct 从未被创建；而且梯度提升前面不需要缩放器。

# 9) Engineering + scaling + model in one pipeline (the engineering step learns from TRAIN only)
import joblib
from sklearn.base import BaseEstimator, TransformerMixin
from sklearn.pipeline import Pipeline
from sklearn.model_selection import cross_val_score
class LoanFeatures(BaseEstimator, TransformerMixin):
    def fit(self, X, y=None):
        self.age_mean_ = X["age"].mean()                               # learned from training rows
        return self
    def transform(self, X):
        out = pd.DataFrame(index=X.index)
        out["debt_to_income"] = X["total_debt"] / (12 * X["monthly_income"].where(X["monthly_income"] > 0))
        out["age_dev_sq"] = (X["age"] - self.age_mean_) ** 2            # centred square: U-shape, low collinearity
        out[["age", "credit_score", "years_employed"]] = X[["age", "credit_score", "years_employed"]]
        return out
    def get_feature_names_out(self, input_features=None):
        return np.array(["debt_to_income", "age_dev_sq", "age", "credit_score", "years_employed"])
raw_cols = ["age", "monthly_income", "total_debt", "credit_score", "years_employed"]
pipe = Pipeline([("features", LoanFeatures()), ("scale", StandardScaler()), ("model", LogisticRegression())])
cv = cross_val_score(pipe, train[raw_cols], y_tr, cv=5, scoring="roc_auc")
pipe.fit(train[raw_cols], y_tr)
print(f"CV AUC {cv.mean():.3f} ± {cv.std():.3f} | test AUC {roc_auc_score(y_te, pipe.predict_proba(test[raw_cols])[:, 1]):.3f}")
print("model sees:", pipe[:-1].get_feature_names_out().tolist())
joblib.dump(pipe, "loan_pipeline.joblib")
new_applicant = pd.DataFrame([{"age": 24, "monthly_income": 38000, "total_debt": 410000, "credit_score": 655, "years_employed": 1}])
print(f"new applicant → default probability {joblib.load('loan_pipeline.joblib').predict_proba(new_applicant)[0, 1]:.2f}")

# 10) Raw vs engineered vs engineered + scaled, for three kinds of model
from sklearn.neighbors import KNeighborsClassifier
from sklearn.preprocessing import FunctionTransformer
eng = LoanFeatures()
models = {"Logistic": LogisticRegression(max_iter=5000), "KNN (k=50)": KNeighborsClassifier(50),
          "HistGradientBoosting": HistGradientBoostingClassifier(random_state=0)}
rows = []
for name, m in models.items():
    r = {"model": name}
    for label, steps in [("raw", [("m", m)]),
                         ("engineered", [("f", LoanFeatures()), ("m", m)]),
                         ("engineered + scaled", [("f", LoanFeatures()), ("s", StandardScaler()), ("m", m)])]:
        with warnings.catch_warnings():
            warnings.simplefilter("ignore")                                    # unscaled logistic warns about convergence
            p = Pipeline(steps).fit(train[raw_cols], y_tr)
        r[label] = roc_auc_score(y_te, p.predict_proba(test[raw_cols])[:, 1])
    rows.append(r)
bench = pd.DataFrame(rows).set_index("model")
print("test AUC\n", bench.round(3).to_string())

Multicollinearity — The Side Effect of Engineering 🔗Multicollinearity — Engineering का Side Effect 🔗多重共线性 —— 特征构造的副作用 🔗

When you add debt ÷ income next to debt and income, or age² next to age, the new columns are strongly related to the old ones. This is multicollinearity. It usually does not hurt predictions. It hurts explanations: in linear and logistic models, the weight can move between related columns, so individual coefficients become unstable and their signs can flip. The Variance Inflation Factor (VIF) measures how well each column is predicted by the others: about 1 is fine, above 5 deserves a look, above 10 is strong.

जब आप debt और income के साथ debt ÷ income, या age के साथ age² जोड़ते हैं, तो नए columns पुराने से गहरे जुड़े होते हैं। यह multicollinearity है। यह आमतौर पर predictions को नुक़सान नहीं पहुँचाता। यह व्याख्या को नुक़सान पहुँचाता है: linear और logistic models में वज़न जुड़े columns के बीच खिसक सकता है, इसलिए अलग-अलग coefficients अस्थिर हो जाते हैं और उनके signs पलट सकते हैं। Variance Inflation Factor (VIF) मापता है कि हर column बाक़ी से कितनी अच्छी तरह predict होता है: लगभग 1 ठीक है, 5 से ऊपर देखने लायक, 10 से ऊपर मज़बूत।

当你把负债 ÷ 收入和负债、收入放在一起，或把 age² 和 age 放在一起时，新列与旧列高度相关。这就是多重共线性。它通常不影响预测，影响的是解释：在线性和逻辑模型中，权重可以在相关列之间转移，单个系数因而不稳定，甚至变号。方差膨胀因子（VIF）衡量每列能被其他列预测的程度：约 1 没问题，超过 5 值得检查，超过 10 属于严重。

# 8) Multicollinearity: VIF (with the constant!) and unstable coefficients
from statsmodels.stats.outliers_influence import variance_inflation_factor
train = train.assign(income_sq=train["monthly_income"] ** 2); test = test.assign(income_sq=test["monthly_income"] ** 2)
mc = ["monthly_income", "income_sq", "total_debt", "debt_to_income", "age", "age_sq", "credit_score", "years_employed"]
import warnings
with warnings.catch_warnings(record=True) as w:
    warnings.simplefilter("always")
    old = [variance_inflation_factor(train[mc].values, i, standardize=False) for i in range(len(mc))]   # pre-0.15 behaviour, no constant
vif = pd.DataFrame({"VIF": [variance_inflation_factor(train[mc], i) for i in range(len(mc))],          # statsmodels ≥ 0.15 standardises for you
                    "old way, no constant": old}, index=mc).round(1)
print(vif.to_string())
print("statsmodels warned:", str(w[0].message).split(". ")[0] if w else "nothing")
fixed = ["debt_to_income", "age", "age_sq", "credit_score", "years_employed"]
vif_of = lambda d: {c: round(float(variance_inflation_factor(d[fixed], i)), 1) for i, c in enumerate(fixed)}
print("\nafter dropping income, income² and debt:", vif_of(train))
age_c = train["age"].mean()
cen = train.assign(age_sq=(train["age"] - age_c) ** 2)
print("…and centring age before squaring:      ", vif_of(cen))
# bootstrap: how stable is the income coefficient?
r = np.random.default_rng(0)
def coef_spread(cols, C):
    out = []
    for b in range(100):
        i = r.integers(0, len(train), len(train))
        m = make_pipeline(StandardScaler(), LogisticRegression(C=C, max_iter=5000)).fit(train[cols].iloc[i], y_tr.iloc[i])
        out.append(m[-1].coef_[0][cols.index("monthly_income")])
    return np.percentile(out, [5, 95]), np.mean(np.sign(out) > 0)
for label, cols, C in [("income + income² + debt + ratio, almost no penalty", mc, 1e6),
                       ("same columns, L2 penalty C = 0.05", mc, 0.05),
                       ("income + ratio only, almost no penalty", ["monthly_income", "debt_to_income", "age", "age_sq", "credit_score", "years_employed"], 1e6)]:
    (lo, hi), pos = coef_spread(cols, C)
    print(f"{label:<52} income coef 90% range [{lo:+.2f}, {hi:+.2f}], positive in {pos:.0%} of fits")
for cols, C in [(mc, 1e6), (fixed, 1e6)]:
    print(f"test AUC with {len(cols)} columns: {auc(make_pipeline(StandardScaler(), LogisticRegression(C=C, max_iter=5000)), cols):.3f}")

Make the Practice Data 🐍Practice Data बनाएँ 🐍生成练习数据 🐍

Run this first. In the loans, default depends on debt ÷ yearly income, a U-shaped age effect, credit score and years employed — not on income itself. In the orders, delivery time depends on distance, a night-time slowdown that wraps around midnight, Fridays, weekends and each customer's area.

इसे पहले चलाएँ। Loans में default debt ÷ सालाना income, U-आकार के age असर, credit score और नौकरी के सालों पर निर्भर है — ख़ुद income पर नहीं। Orders में delivery समय distance, आधी रात के दोनों ओर फैली रात की धीमी गति, Friday, weekend और हर ग्राहक के इलाक़े पर निर्भर है।

请先运行它。在贷款数据中，违约取决于负债 ÷ 年收入、U 形的年龄效应、信用分和工作年限 —— 而不取决于收入本身。在订单数据中，配送时间取决于距离、跨越午夜的夜间减速、周五、周末以及每位客户所在区域。

# 0) Two practice tables: loans (for ratios, powers, bins, collinearity) and delivery orders (for dates and group features)
import numpy as np, pandas as pd
rng = np.random.default_rng(21)
n = 4000
age = rng.integers(21, 71, n)
income = np.round(np.exp(rng.normal(10.9, 0.55, n)), -2)                 # ₹ per month, long right tail
dti = np.clip(rng.gamma(3, 0.12, n), 0.02, 1.6)                          # debt ÷ yearly income (the real driver)
debt = np.round(dti * income * 12, -3)                                   # ₹ outstanding
credit_score = np.round(np.clip(rng.normal(710, 55, n), 450, 900))
years_employed = np.clip(np.round(rng.gamma(2, 3, n)), 0, age - 20)
z = -1.2 + 4.0 * (dti - 0.36) + 0.0022 * (age - 45) ** 2 - 0.012 * (credit_score - 710) - 0.04 * years_employed
default = (rng.random(n) < 1 / (1 + np.exp(-z))).astype(int)
pd.DataFrame({"age": age, "monthly_income": income, "total_debt": debt, "credit_score": credit_score,
              "years_employed": years_employed, "default": default}).to_csv("loans.csv", index=False)
print(f"loans.csv : {n:,} rows, default rate {default.mean():.1%}")

m, n_cust = 8000, 600
cust = rng.integers(0, n_cust, m)
cust_area = rng.normal(0, 3.0, n_cust)                                   # some customers live in slow areas (hours)
ts = pd.Timestamp("2025-01-01") + pd.to_timedelta(np.sort(rng.uniform(0, 365 * 24, m)), unit="h")
hour, dow = ts.hour.to_numpy(), ts.dayofweek.to_numpy()
distance = np.round(rng.gamma(2.2, 4, m), 1)
night = np.cos(2 * np.pi * (hour - 1) / 24)                              # slowest around 1 AM, fastest around 1 PM
hours = (6 + 0.35 * distance + 4.0 * night + 3.0 * (dow == 4) + 2.0 * (dow >= 5)
         + cust_area[cust] + rng.normal(0, 2.5, m))
pd.DataFrame({"order_id": np.arange(1, m + 1), "customer_id": cust + 1001, "order_time": ts,
              "distance_km": distance, "delivery_hours": np.round(np.clip(hours, 0.5, None), 2)}).to_csv("orders.csv", index=False)
print(f"orders.csv: {m:,} orders from {n_cust} customers, {ts.min():%d %b %Y} – {ts.max():%d %b %Y}")

Golden Rules of Feature Engineering 🏆Feature Engineering के सुनहरे नियम 🏆特征工程黄金法则 🏆