Feature Engineering & Feature Scaling
Press Next → or use ← → arrow keys
Why Feature Engineering Matters
Create, transform and combine features first, then scale the result. Scale first and your ratios become dimensionless nonsense that no longer mean anything.
Six Ways to Engineer a Feature
Ask what an expert mentally computes when making the same call — a credit officer thinks "debt-to-income", a doctor thinks "BMI". Encode that intuition and the model inherits it.
Interaction Features — Ratios & Products
A microfinance model jumped from AUC 0.71 to 0.84 on one feature: debt_to_income. Debt and income alone were weak — their ratio was the single most predictive signal in the whole dataset.
# ratios — always guard the divisor
df['debt_to_income'] = \
df['debt'] / (df['income'] + 1)
df['spend_per_day'] = \
df['amount'] / (df['days'] + 1)
# products & differences
df['price_x_qty'] = df['price']*df['qty']
df['balance_change'] = \
df['end'] - df['start']
# safe divide
df['r'] = np.where(df['den'] > 0,
df['num']/df['den'], 0)Polynomial Features — Let Lines Bend
poly = PolynomialFeatures(degree=2)
# age, income → +age², age·income, income²
Pipeline([
('scaler', StandardScaler()), # FIRST
('poly', PolynomialFeatures(2)),
('model', Ridge(alpha=1.0))]) # not OLS!Expanding unscaled values gives astronomical x² terms and numerical chaos. And poly terms are always correlated by construction — use Ridge/Lasso, never plain OLS.
Date/Time & Cyclical Encoding
A delivery model on raw Unix timestamps missed that Friday drops take 40% longer. Extracting is_friday, day_of_week and hour cut RMSE from 4.2 to 1.8 hours.
df['hour_sin'] = np.sin(2*np.pi*df['hour']/24); df['hour_cos'] = np.cos(2*np.pi*df['hour']/24)
df['is_weekend'] = (df['order_date'].dt.dayofweek >= 5).astype(int)Binning — Equal-Width vs Equal-Frequency
df['age_grp'] = pd.cut(df['age'], bins=[0,25,35,45,55,100], labels=['18-25','26-35','36-45','46-55','55+'])
df['inc_q'] = pd.qcut(df['income'], q=4, labels=['Q1','Q2','Q3','Q4']) # better for skewAggregation — Signals a Single Row Can't See
# customer-level stats → merge back
agg = df.groupby('customer_id').agg(
avg_spend=('amount', 'mean'),
order_count=('order_id', 'count'),
return_rate=('is_returned', 'mean'),
).reset_index()
df = df.merge(agg, on='customer_id')
# rolling window (time series)
df['roll_30d'] = (df.groupby('customer_id')
['amount'].transform(
lambda x: x.rolling(30,
min_periods=1).mean()))A customer's average spend, order count and return rate say far more about their next action than any one order. Aggregations inject that history into every row.
Compute group stats on the training set, then merge onto test with those same stats — computing on the full data leaks the future into the past.
Domain Features — Encoding Expertise
| Domain | Raw | Engineered | Why it works |
|---|---|---|---|
| Credit | debt, income | debt / income | Affordability — the standard risk metric |
| E-commerce | spend, visits | spend / visits | Value per engagement → premium behaviour |
| Healthcare | weight, height | weight / height² | BMI — an established medical indicator |
| Retail | last_purchase, today | days_since_last | Recency — the strongest repeat-purchase signal |
| Telecom | calls, plan_limit | calls / limit × 100 | Utilisation % → predicts upsell & churn |
| Finance | assets, liabilities | assets / liabilities | Current ratio — standard liquidity metric |
These aren't clever tricks — they're the numbers domain experts already trust. Encoding them hands the model decades of accumulated professional judgment for free.
Then — and Only Then — Scale
| Scaler | Formula | Range | Best for |
|---|---|---|---|
| MinMax | (x−min)/(max−min) | [0, 1] | Neural nets, image data |
| Standard | (x−μ)/σ | μ=0, σ=1 | Linear models, SVM, PCA — the default |
| Robust | (x−Q2)/IQR | median-centred | Legitimate outliers present |
| MaxAbs | x/|max| | [−1, 1] | Sparse data, TF-IDF (keeps zeros) |
clean → engineer → encode → scale → select. Scaling must come after engineering so your ratios, powers and aggregations all end up on a comparable footing before the model ever sees them.
Where the Accuracy Actually Comes From
The largest jump comes from engineering, especially for linear models. Scaling then adds more for distance- and gradient-based algorithms. Trees skip the scaling gain but still love the new features.
One Pipeline: Engineer → Scale → Model
class FeatureEngineer(BaseEstimator,
TransformerMixin):
def fit(self, X, y=None): return self
def transform(self, X):
X = X.copy()
X['debt_to_income'] = X.debt/(X.income+1)
X['hour_sin'] = np.sin(2*np.pi*X.hour/24)
return X
full = Pipeline([
('engineer', FeatureEngineer()),
('prep', preprocessor), # std/robust/mm/ohe
('model', GradientBoostingClassifier())])
full.fit(X_train, y_train)
joblib.dump(full, 'fe_pipeline.pkl')The Hidden Danger — Multicollinearity
annual_income had a VIF over 45 from
correlating with debt_to_income and income_sq, making the
coefficients mathematically unstable. Dropping the twins and using Ridge flipped it to a sane +1.4.
Who Multicollinearity Hurts
| Algorithm | Harmed? | Fix |
|---|---|---|
| Linear / Logistic Regression | severely | Drop features or use Ridge/L2 |
| Ridge / Lasso | partly | Already mitigated — Lasso auto-selects |
| KNN | yes | Drop duplicated-signal features |
| SVM | mildly | Apply PCA before SVM |
| Decision Tree / RF / XGBoost | no | Nothing needed — splits ignore it |
| Neural Network | mildly | Drop correlated pairs (faster convergence) |
VIF < 5 acceptable · 5–10 investigate · > 10 serious multicollinearity. It answers "how well do the other features predict this one?" — high means redundant.
Four Ways to Fix It
Any PolynomialFeatures(degree≥2) produces terms correlated by maths — never follow it with plain OLS. Always Ridge or Lasso.
The Golden Rules — 1 to 5
num / (den + 1) or np.where(den>0, …) — one zero can poison a column.The Golden Rules — 6 to 9 & Takeaway
BaseEstimator + TransformerMixin) so it replays identically at inference.The ₹18-crore unlock came from three features, not a new architecture. Feature engineering — where domain knowledge, creativity and maths meet — is the highest-leverage step in the whole workflow. Invest there first.
🛠️ End of tutorial · Press ← to review, or click Restart