Exploratory Data Analysis with Pandas
Press Next → or use ← → arrow keys
The Intuition — Meet Your Data First
Exploratory Data Analysis (EDA) is that walk-through for a dataset. Before any model, you read the data, understand its structure and types, compute summary statistics, and look for patterns, gaps, and oddities — so the questions you ask next are the right ones.
EDA has three goals: understand the structure (rows, columns, types), assess quality (missing values, duplicates, outliers), and surface patterns (distributions and relationships). It turns a raw table into a mental model you can actually reason about.
The EDA Workflow
EDA isn't a straight line — a surprising distribution sends you back to check quality; a suspicious correlation prompts a new plot. You cycle through inspect → question → visualize until the dataset holds no more surprises. The output is a set of hypotheses worth testing, not a final answer.
The First Five Commands
import pandas as pd df = pd.read_csv("data.csv") df.head() # first 5 rows — what does a record look like? df.shape # (rows, columns) — how big is it? df.info() # dtypes + non-null counts — types & missingness at a glance df.dtypes # is 'age' really numeric, or read as object? df.describe() # count, mean, std, min, quartiles, max (numeric cols)
These five calls answer the essentials in seconds: what a row looks like (head), how large
the data is (shape), which columns have missing values and wrong types (info,
dtypes), and the shape of every numeric column (describe). Never skip this — a
column silently read as text instead of a number breaks everything downstream.
describe(include='object') For Categories
Plain describe() covers only numeric columns. Pass include='object' (or
'all') to see count, unique, top, and frequency for text/categorical columns too.
Summarizing Every Column
normalize=True for proportions.df.groupby('region')['sales'].mean() compares a metric across groups.df['category'].value_counts() # counts per category df['category'].value_counts(normalize=True) # proportions df.groupby('region')['revenue'].agg(['mean', 'median', 'count'])
Missing Values & Duplicates
df.isnull().sum() counts missing values per column; df.duplicated().sum()
catches repeated rows. Here income is the worst offender (3 gaps). Don't fix them yet —
EDA's job is to map the damage so the cleaning step knows where to focus.
Univariate → Bivariate → Multivariate
Univariate studies one column at a time (histograms, value_counts). Bivariate pairs two (scatter, groupby, a single correlation). Multivariate looks at many together (correlation matrix, pairplot). Work up the ladder — each level answers a bigger question.
One Column, Fully Understood
A quick df['col'].hist() reveals the shape — here a right skew — while
df['col'].plot.box() exposes the spread and outliers. For categorical
columns, value_counts().plot.bar() does the same job. Univariate analysis answers: is it
skewed? bimodal? riddled with outliers? capped?
How Two Columns Relate
df['a'].corr(df['b']). Shows direction, strength, and non-linearity.groupby('cat')['num'].mean() or a box plot per category — compares distributions across groups.pd.crosstab(a, b) tallies co-occurrence; normalize it for conditional proportions.df.groupby('plan')['monthly_spend'].mean() # numeric by category pd.crosstab(df['region'], df['churned']) # category vs category df['age'].corr(df['income']) # one correlation
Everything At Once
# correlation matrix across all numeric columns corr = df.corr(numeric_only=True) import seaborn as sns sns.heatmap(corr, annot=True, cmap='coolwarm') # the whole picture sns.pairplot(df, hue='target') # every pair scattered
One df.corr() heatmap surfaces every strong relationship at a glance: which features track
your target (keep them), and which features mirror each other (redundant — a multicollinearity
warning). A pairplot then lets you eyeball each pair's actual shape, catching the
non-linear patterns a single correlation number would miss.
Adding hue='target' to a pairplot colours points by class — instantly showing which
features actually separate your groups, a huge head-start on feature selection.
Which Plot For Which Question?
| Question | Plot | pandas / seaborn |
|---|---|---|
| Shape of one numeric column | Histogram | df['x'].hist() |
| Spread & outliers of one column | Box plot | df['x'].plot.box() |
| Counts of a category | Bar chart | value_counts().plot.bar() |
| Relationship of two numerics | Scatter | df.plot.scatter('a','b') |
| A numeric across categories | Grouped box | sns.boxplot(x,y) |
| All pairwise relationships | Heatmap / pairplot | sns.heatmap(df.corr()) |
Summary statistics can hide skew, outliers, and non-linearity — remember Anscombe's Quartet. Pair every
describe() with a plot. A five-second histogram catches problems a table of means would let
slip straight into your model.
EDA Traps To Sidestep
| ❌ The Trap | ✅ The Fix |
|---|---|
| Skipping straight to modelling | Always inspect + plot first — GIGO applies here too |
Trusting describe() alone | Visualize; stats hide skew & outliers |
| Ignoring dtypes (numbers read as text) | Check df.dtypes; cast early |
| Treating correlation as causation | EDA raises hypotheses, doesn't prove them |
| Cleaning during exploration | Map issues now; fix them in the cleaning step |
EDA's output is a map of the data and a list of questions — not final answers. A striking pattern is a hypothesis to test later, not a proven fact. Keeping that mindset stops you from fooling yourself early.
Six Rules For EDA
You Can Now Interrogate Any Dataset
EDA is the disciplined first look: load the data, inspect its structure and types, map its quality problems, then climb from single columns to relationships — always pairing statistics with plots. Done well, it hands the rest of your pipeline a clear map and a sharp set of questions.
EDA surfaces the problems; data cleaning fixes them — missing values, duplicates, outliers, and type errors. After that comes feature engineering and transformation, turning your now-understood data into model-ready inputs.
🔍 End of tutorial · Press ← to review, or click Restart