Machine Learning 📂 Supervised Learning · 11 of 19 206 min read

Naïve Bayes Classifier

Learn how Naïve Bayes uses Bayes' Theorem and the conditional independence assumption to classify text, detect spam, and diagnose disease — fast, interpretable, and surprisingly accurate even on tiny datasets.

🌐 Language
▶️
This Tutorial Has Seven Simulators

Look for the interactive boxes: Bayes the Intern (Section 01), Bayes’ Theorem with 1000 Emails (02), Underflow (03), Hand Calculator (04), Spam Filter Playground (05), Gaussian NB Explorer (07) and Double Counting (12). Every number in them is computed live in your browser.

Section 01

The Story That Explains Naïve Bayes

📖 Real-World Analogy
The New Intern Who Learned to Spot Spam
You hire a new intern called Bayes to sort your inbox. On his first day you give him 1,000 old emails — 400 spam and 600 normal (“ham”) — and ask him to take notes. He reads every email and keeps a simple tally:

“The word FREE appears in 380 spam emails but only 12 ham emails.”
“The word meeting appears in 420 ham emails but only 3 spam emails.”
“The word winner appears in 350 spam emails but only 2 ham emails.”

A new email arrives: “You are a FREE WINNER — claim your prize now!”

Bayes does not read it like a human. He asks one cold question: “Given these words, how likely is it that this email is spam?” He starts from how common spam is (40%), multiplies in the evidence from each word, compares Spam with Ham, and says: “99.99% spam.” Correct — deleted in milliseconds.

That is Naïve Bayes. It is called “naïve” because it assumes every word gives its evidence independently of the others. That assumption is usually false, but the method still works remarkably well.
🔍
Why “Naïve”?

In real emails, “FREE” and “WINNER” are not independent — if you see one, the other becomes more likely. Naïve Bayes ignores all such links and treats every feature as a separate, independent clue. Statistically wrong. Practically powerful — especially for text.

Try It — Bayes the Intern

Click words in Bayes’s notebook, or play one of the example emails. Watch each word push the odds towards spam or ham.

🧑‍💼 Bayes the InternHis notebook: 400 spam and 600 ham emails. Click words to add them to a new email.
prior P(spam) before reading
each bar = how much one word changes the odds   → towards spam   ← towards ham
Section 02

The Foundation — Bayes’ Theorem

Everything in Naïve Bayes comes from one equation by the Reverend Thomas Bayes (18th century). It tells you how to update a belief when you see new evidence.

Bayes’ Theorem
P(A | B) = P(B | A) × P(A) / P(B)
The probability of A given B = the likelihood of B when A is true × the prior belief in A ÷ the overall probability of B.
In Classification Terms
P(Class | Features) = P(Features | Class) × P(Class) / P(Features)
How likely is this class, given the feature values we see in this sample?
Term Name Plain English Spam Example
P(Class | Features) Posterior What we want — the probability of the class after seeing the evidence P(Spam | “FREE”, “WIN”)
P(Features | Class) Likelihood How probable are these features if we already know the class? P(“FREE”, “WIN” | Spam)
P(Class) Prior How common the class is in the training data, before looking at any features P(Spam) = 400/1000 = 0.40
P(Features) Evidence How common these features are overall. It is the same for every class Only needed to turn scores into probabilities
💡
The Denominator Trick — the MAP Decision Rule

When we compare classes (Spam vs Ham), P(Features) is the same for both. So to decide, we only compare P(Features | Class) × P(Class) and pick the largest. This is the Maximum A Posteriori (MAP) rule. If we also want a probability (like “97% spam”), we divide each score by the sum of all scores.

Try It — Bayes’ Theorem with 1000 Emails

Bayes’ theorem is just counting. Press Show only emails with FREE and count the red and green squares. Then try the “rare spam” preset to see why the prior matters.

🎯 Bayes’ Theorem with 1000 EmailsEach dot is one email. Squares contain the word “FREE”.
P(spam) — prior
P(FREE | spam)
P(FREE | ham)
■ spam with FREE   ■ ham with FREE   ● spam / ● ham without FREE
Section 03

The Naïve Independence Assumption

With many features, estimating P(word₁, word₂, word₃, … | Class) directly is impossible — you would need to see every exact combination of words in the training data. Naïve Bayes solves this with one bold simplification.

🔑 The Independence Assumption — Features Vote Separately
Without It
P(“free”, “win”, “prize” | Spam) — we must have seen this exact 3-word combination in training. With 10,000 words there are far too many combinations; most were never seen.
With It
P(“free” | Spam) × P(“win” | Spam) × P(“prize” | Spam) — estimate each word on its own and multiply. Always computable.
Result
The joint probability becomes a simple product. Training is just counting, so it is very fast — and surprisingly accurate in many real applications.
General MAP Formula
ŷ = argmax P(Cₖ) ∏ P(xᵢ | Cₖ)
Choose the class Cₖ with the largest product of its prior and the likelihood of each feature.
Log-Space Version (Numerically Stable)
ŷ = argmax [log P(Cₖ) + Σ log P(xᵢ | Cₖ)]
The logarithm turns multiplication into addition, so many tiny probabilities do not underflow to zero.
⚠️
Why Log-Space Is Not Optional

Multiplying hundreds of small probabilities — 0.001 × 0.003 × 0.0005 × … — gives numbers so tiny that the computer silently stores them as zero (floating-point underflow). Logarithms turn products into sums: log(a × b) = log(a) + log(b). Because log always increases, the class with the biggest log-score is also the class with the biggest product. scikit-learn does this for you, but you must do it yourself when you write Naïve Bayes from scratch.

Try It — Underflow

Press Play and watch both products fall towards the red zone. Once they hit it, the raw products are both 0 — but the log-scores still give the right answer.

🧊 Underflow: Why Log-Space?Every word has P(word | spam) = p and P(word | ham) = 0.8·p. Spam should always win.
number of words in the document
word probability p (log scale)
Section 04

Step-by-Step Manual Calculation

Let us classify one new email by hand. We use a tiny training set of 10 emails with three yes/no features.

# Contains “free” Contains “winner” Contains “meeting” Label
1✅✅❌Spam
2✅❌❌Spam
3✅✅❌Spam
4❌✅❌Spam
5✅❌❌Spam
6❌❌✅Ham
7❌❌✅Ham
8❌❌✅Ham
9✅❌✅Ham
10❌❌❌Ham

New email to classify: contains “free” ✅, “winner” ✅, “meeting” ❌. Spam or Ham?

🧮 Step 1 — Prior Probabilities
P(Spam)
5 spam emails out of 10 → P(Spam) = 5/10 = 0.500
P(Ham)
5 ham emails out of 10 → P(Ham) = 5/10 = 0.500
🧮 Step 2 — Likelihoods for Each Class
P(free|Spam)
4 of 5 spam emails contain “free” (1, 2, 3, 5) → 4/5 = 0.800
P(winner|Spam)
3 of 5 spam emails contain “winner” (1, 3, 4) → 3/5 = 0.600
P(¬meeting|Spam)
5 of 5 spam emails do not contain “meeting” → 5/5 = 1.000
P(free|Ham)
1 of 5 ham emails contains “free” (9) → 1/5 = 0.200
P(winner|Ham)
0 of 5 ham emails contain “winner” → 0/5 = 0.000 ⚠️ Zero problem!
P(¬meeting|Ham)
Only 1 of 5 ham emails does not contain “meeting” (10) → 1/5 = 0.200
🧮 Step 3 — Multiply to Get the MAP Scores
Spam
P(Spam) × P(free|Spam) × P(winner|Spam) × P(¬meeting|Spam)
= 0.5 × 0.800 × 0.600 × 1.000 = 0.240
Ham
P(Ham) × P(free|Ham) × P(winner|Ham) × P(¬meeting|Ham)
= 0.5 × 0.200 × 0.000 × 0.200 = 0.000 ← one zero kills it!
Decision
0.240 > 0.000 → Predicted: SPAM ✅ — correct, but the zero is a serious flaw (see Section 06)
🎯
Correct — But One Fatal Issue

The model predicted spam correctly. But the Ham score is exactly zero, because “winner” never appeared in a ham email during training. One unseen word cancelled all the other evidence for ham. This is the Zero Frequency Problem. Laplace smoothing fixes it in Section 06.

Try It — Hand Calculator

The same table, but live. Every factor is shown as a fraction. Click cells to change the training data, change the new email, and move α from 0 to 1 to see smoothing at work.

🧮 Hand Calculator — the 10-Email TableClick any cell to change the training data. Then move α.
smoothing α
new email:
Section 05

The Variants of Naïve Bayes

Naïve Bayes is not one algorithm — it is a family. Choose the variant by the type of features in your data.

📊
Gaussian NB
Continuous (Real-Valued) Features
Assumes each feature follows a normal (bell-shaped) distribution inside each class. It learns the mean μ and variance σ² of every feature for every class, then scores new samples with the Gaussian density. Used for measurements: sensors, lab values, physical sizes.
✅ Handles continuous data directly
❌ Weak when features are far from bell-shaped
📝
Multinomial NB
Count / Frequency Features
Made for word counts (TF-IDF values also work well in practice). It models how often each word appears in each class. A classic choice for spam filtering, news topics and document categories. Features must be non-negative.
✅ Strong, fast baseline for text
❌ Cannot use negative feature values
🔘
Bernoulli NB
Binary (0 / 1) Features
Uses presence or absence of each word (1 = in the email, 0 = not). Important: it also uses absent words as evidence — Multinomial ignores them. Often good on short texts, where whether a word appears matters more than how often.
✅ Good for short texts and binary features
❌ Throws away how often a word appears
Variant Feature Type Likelihood Model Typical Use Cases
Gaussian NB Real-valued, continuous Gaussian density — uses μ, σ² Iris, medical measurements, sensors
Multinomial NB Non-negative counts (or TF-IDF) Multinomial distribution over word counts Spam, news topics, document tagging
Bernoulli NB Binary 0 / 1 One yes/no (Bernoulli) probability per feature Short texts, binary feature sets
Complement NB Non-negative counts (or TF-IDF) Learns from the complement of each class Imbalanced text classification
Categorical NB Categories (colour, city…) One probability per category value Survey answers, encoded categories

Try It — Spam Filter Playground

This simulator uses word counts from the real SMS Spam Collection (5,572 messages). Type any message, switch between Multinomial and Bernoulli, and see which words push towards spam.

💬 Spam Filter PlaygroundTrained on 5,572 real SMS messages (747 spam). Type any message.
α (log scale)
+ pushes towards spam   − pushes towards ham   (log-likelihood ratio)   struck-out words are not in the 600-word vocabulary
Section 06

The Zero Frequency Problem — Laplace Smoothing

📖 The Problem
One Missing Word Destroys Everything
Our intern Bayes has a serious flaw. If a word in a new email was never seen in any training email of a class, the probability for that whole class becomes exactly zero — no matter how strongly all the other words point to it. It is like a jury ignoring ten witnesses because an eleventh witness did not show up. In the real SMS data, “prize” and “claim” never appear in a single ham message — so without smoothing, any message with “prize” could never be ham.
Without Smoothing (Broken)
P(x | Class) = count / total
If count = 0 → probability = 0 → the whole product = 0. One unseen word cancels every other feature.
Laplace Smoothing — Bernoulli (yes/no features)
P(word present | C) = (docs in C with word + α) / (docs in C + 2α)
Add α imaginary emails for each of the 2 outcomes (present / absent). α = 1 is Laplace smoothing.
Laplace Smoothing — Multinomial (word counts)
P(word | C) = (count of word in C + α) / (total words in C + α × V)
V is the vocabulary size. Add α to the count of every word, so no probability can ever be zero.
🧮 Laplace Smoothing (α = 1) Applied to Our 10 Emails
Rule
Yes/no features, so each factor = (count + 1) / (5 + 2). We smooth every factor, not only the zero one.
Spam
free (4+1)/7 = 0.714  |  winner (3+1)/7 = 0.571  |  ¬meeting (5+1)/7 = 0.857
Score = 0.5 × 0.714 × 0.571 × 0.857 = 0.1749
Ham
free (1+1)/7 = 0.286  |  winner (0+1)/7 = 0.143 ✅ no longer zero  |  ¬meeting (1+1)/7 = 0.286
Score = 0.5 × 0.286 × 0.143 × 0.286 = 0.0058
Final
P(Spam | email) = 0.1749 / (0.1749 + 0.0058) = 96.8% → Still SPAM ✅ — but now Ham has a fair, non-zero score

scikit-learn’s BernoulliNB gives exactly the same numbers:

import numpy as np
from sklearn.naive_bayes import BernoulliNB

# The 10 training emails from Section 04: [free, winner, meeting]
X = np.array([[1,1,0],[1,0,0],[1,1,0],[0,1,0],[1,0,0],   # spam
              [0,0,1],[0,0,1],[0,0,1],[1,0,1],[0,0,0]])  # ham
y = np.array(["spam"]*5 + ["ham"]*5)
new_email = [[1, 1, 0]]          # has "free", has "winner", no "meeting"

for alpha in [1e-10, 1.0]:       # almost no smoothing, then Laplace
    nb = BernoulliNB(alpha=alpha, force_alpha=True).fit(X, y)
    p = nb.predict_proba(new_email)[0]
    print(f"alpha={alpha:g}:  P(ham)={p[0]:.4f}  P(spam)={p[1]:.4f}  ->  {nb.predict(new_email)[0]}")

# Learned P(word present | class) with alpha = 1
print("\nP(word | class), alpha=1:")
for cls, row in zip(nb.classes_, np.exp(nb.feature_log_prob_)):
    print(f"  {cls:4s}  free={row[0]:.3f}  winner={row[1]:.3f}  meeting={row[2]:.3f}")
Output
alpha=1e-10: P(ham)=0.0000 P(spam)=1.0000 -> spam alpha=1: P(ham)=0.0323 P(spam)=0.9677 -> spam P(word | class), alpha=1: ham free=0.286 winner=0.143 meeting=0.714 spam free=0.714 winner=0.571 meeting=0.143
🎛️
Tuning Alpha (α)

α = 1 → Laplace smoothing (add-one). 0 < α < 1 → Lidstone smoothing (gentler). α = 0 → no smoothing (dangerous — scikit-learn even replaces it with a tiny number like 1e-10 unless you set force_alpha=True). The default in scikit-learn is alpha=1.0. Tune α between about 0.01 and 2 with cross-validation. Large datasets usually prefer small α; tiny datasets need more smoothing.

Section 07

Gaussian NB — How Continuous Features Are Modelled

When features are continuous (age, glucose level, temperature), counting makes no sense. Gaussian NB instead models each feature as a bell curve inside each class — it estimates the mean μ and variance σ² from the training data.

Patient Age Glucose (mg/dL) BMI Diabetes?
15018033.6Yes
25820035.0Yes
34516530.5Yes
4259022.1No
5308523.5No
62810024.0No
📐 Training — Estimate μ and σ for Each Class (as scikit-learn does)
Diabetic
Age: μ = 51.0, σ = 5.4  |  Glucose: μ = 181.7, σ = 14.3  |  BMI: μ = 33.0, σ = 1.9
Healthy
Age: μ = 27.7, σ = 2.1  |  Glucose: μ = 91.7, σ = 6.2  |  BMI: μ = 23.2, σ = 0.8
New patient
Age = 40, Glucose = 150, BMI = 29. Which class?
Log-scores
Diabetic: log 0.5 + (−4.71 − 6.02 − 3.85) = −15.3  |  Healthy: log 0.5 + (−19.65 − 46.50 − 26.71) = −93.6
Prediction
The diabetic score is far higher → Diabetic. The patient’s values are between the two groups, but much closer to the diabetic ones when measured in σ units.
⚠️
Note on σ

scikit-learn divides by n (not n − 1) when it computes the variance, so its σ values are a bit smaller than a textbook “sample” standard deviation. With only 3 patients per class the σ values are tiny, so the model becomes extremely sure (here P(diabetic) ≈ 100%). Real models need far more data.

Gaussian PDF
P(x | Class) = (1 / σ√2π) × e^(−(x−μ)² / 2σ²)
The density of value x for a class with mean μ and standard deviation σ.
Full Gaussian NB Prediction
ŷ = argmax [log P(Cₖ) + Σ log PDF(xᵢ ; μₖᵢ, σₖᵢ)]
Add the log-density of every feature to the log-prior for each class. Pick the class with the highest total.

Try It — Gaussian NB Explorer

Click anywhere to place a new patient and see the bell curves and the log-score sums. Then try the three data shapes — the last one shows exactly where the naïve assumption breaks.

🔔 Gaussian NB Explorer45 healthy and 45 diabetic patients. Dashed ellipses = what Gaussian NB learns (±2σ per feature).
● healthy   ■ diabetic   shaded = Gaussian NB prediction   ★ your patient
Section 08

Python Implementation

Gaussian Naïve Bayes — Iris Classification

from sklearn.naive_bayes import GaussianNB
from sklearn.model_selection import train_test_split
from sklearn.metrics import classification_report
from sklearn.datasets import load_iris

# Iris: 4 continuous features -> Gaussian NB is a natural choice
iris = load_iris()
X, y = iris.data, iris.target
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42, stratify=y)

gnb = GaussianNB(var_smoothing=1e-9)   # adds 1e-9 x (largest variance) for stability
gnb.fit(X_train, y_train)

y_pred = gnb.predict(X_test)
print(f"Accuracy: {gnb.score(X_test, y_test):.4f}\n")
print(classification_report(y_test, y_pred, target_names=iris.target_names, digits=3))

# The learned mean (theta_) and variance (var_) for each class and feature
for i, cls in enumerate(iris.target_names):
    print(f"{cls}:")
    for j, feat in enumerate(iris.feature_names):
        print(f"  {feat:18s}  mu={gnb.theta_[i, j]:.3f}  var={gnb.var_[i, j]:.4f}")
Output
Accuracy: 0.9667 precision recall f1-score support setosa 1.000 1.000 1.000 10 versicolor 1.000 0.900 0.947 10 virginica 0.909 1.000 0.952 10 accuracy 0.967 30 macro avg 0.970 0.967 0.967 30 weighted avg 0.970 0.967 0.967 30 setosa: sepal length (cm) mu=4.985 var=0.0928 sepal width (cm) mu=3.415 var=0.1553 petal length (cm) mu=1.477 var=0.0252 petal width (cm) mu=0.255 var=0.0130 versicolor: sepal length (cm) mu=5.930 var=0.2216 sepal width (cm) mu=2.750 var=0.0930 petal length (cm) mu=4.252 var=0.1915 petal width (cm) mu=1.320 var=0.0341 virginica: sepal length (cm) mu=6.610 var=0.4574 sepal width (cm) mu=2.980 var=0.1221 petal length (cm) mu=5.580 var=0.3236 petal width (cm) mu=2.040 var=0.0704

96.7% accuracy (29 of 30 test flowers) with no tuning at all. Notice how small the setosa variances are for petal length and width — that is why setosa is so easy to separate.

Multinomial NB — A Tiny Spam Filter

from sklearn.naive_bayes import MultinomialNB
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.pipeline import Pipeline

emails = [
    "Get free money now win prizes",
    "Congratulations you are a winner claim your reward",
    "Free entry in our lucky draw text WIN to 88888",
    "Hi how are you doing today everything okay",
    "Meeting scheduled for 3pm please confirm attendance",
    "Project update attached please review the report",
    "WINNER you have been selected for a cash prize",
    "Lunch tomorrow at the usual place let me know",
    "Urgent your account will be suspended click here now",
    "Can we reschedule the team standup to Friday afternoon",
]
labels = [1, 1, 1, 0, 0, 0, 1, 0, 1, 0]   # 1 = spam, 0 = ham

pipeline = Pipeline([
    ("tfidf", TfidfVectorizer(stop_words="english")),
    ("clf",   MultinomialNB(alpha=1.0)),          # Laplace smoothing
])
pipeline.fit(emails, labels)                       # only 10 emails, so we train on all of them

new_emails = [
    "Claim your free cash prize now",
    "Please confirm the meeting time for Friday",
    "Free lunch at the team meeting tomorrow",
    "Quarterly budget spreadsheet",                # no known words at all
]
for email, prob in zip(new_emails, pipeline.predict_proba(new_emails)):
    label = "SPAM" if prob[1] > 0.5 else "HAM "
    print(f"[{label}] ({prob[1]*100:5.1f}% spam)  {email}")
Output
[SPAM] ( 71.6% spam) Claim your free cash prize now [HAM ] ( 34.9% spam) Please confirm the meeting time for Friday [HAM ] ( 40.0% spam) Free lunch at the team meeting tomorrow [HAM ] ( 50.0% spam) Quarterly budget spreadsheet

With only 10 training emails the model is weak, but it already gets the direction right. The last message has no known words at all, so the model can only return the prior: 50%. Unknown words are simply ignored.

Bernoulli NB — Binary Bag-of-Words

from sklearn.naive_bayes import BernoulliNB
from sklearn.feature_extraction.text import CountVectorizer

# binary=True: each word becomes 1 (present) or 0 (absent)
pipeline_bnb = Pipeline([
    ("bow", CountVectorizer(binary=True, stop_words="english")),
    ("clf", BernoulliNB(alpha=1.0, binarize=None)),  # data is already 0/1
])
pipeline_bnb.fit(emails, labels)
print("Vocabulary size:", len(pipeline_bnb.named_steps["bow"].vocabulary_))

for email, prob in zip(new_emails, pipeline_bnb.predict_proba(new_emails)):
    label = "SPAM" if prob[1] > 0.5 else "HAM "
    print(f"[{label}] ({prob[1]*100:5.1f}% spam)  {email}")
Output
Vocabulary size: 45 [SPAM] ( 98.8% spam) Claim your free cash prize now [HAM ] ( 8.4% spam) Please confirm the meeting time for Friday [HAM ] ( 14.7% spam) Free lunch at the team meeting tomorrow [SPAM] ( 56.0% spam) Quarterly budget spreadsheet

On the same data, BernoulliNB is much more confident — and it calls “Quarterly budget spreadsheet” spam (56%) even though none of its words were seen. Why? Bernoulli also uses absent words: the message lacks typical ham words such as “meeting”, “project” and “confirm”, and that absence counts as evidence.

Section 09

Full Real-World Pipeline — SMS Spam Collection

Now a real dataset: the SMS Spam Collection — 5,572 real text messages, about 13% spam. We tune the pipeline with a grid search and judge it with macro-F1, because accuracy is misleading when one class is rare. 

import pandas as pd
from sklearn.naive_bayes import MultinomialNB
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.model_selection import train_test_split, GridSearchCV
from sklearn.pipeline import Pipeline
from sklearn.metrics import classification_report, confusion_matrix

# SMS Spam Collection: 5,572 real text messages, about 13% spam
url = "https://raw.githubusercontent.com/justmarkham/pycon-2016-tutorial/master/data/sms.tsv"
sms = pd.read_csv(url, sep="\t", header=None, names=["label", "text"])
print(sms["label"].value_counts().to_dict())

X_train, X_test, y_train, y_test = train_test_split(
    sms["text"], sms["label"], test_size=0.25, random_state=42, stratify=sms["label"])

pipe = Pipeline([("tfidf", TfidfVectorizer()), ("clf", MultinomialNB())])
param_grid = {
    "tfidf__ngram_range": [(1, 1), (1, 2)],
    "tfidf__sublinear_tf": [True, False],
    "clf__alpha": [0.01, 0.1, 0.5, 1.0],
}
grid = GridSearchCV(pipe, param_grid, cv=5, scoring="f1_macro", n_jobs=-1)
grid.fit(X_train, y_train)

print("Best params:", grid.best_params_)
print(f"Best CV macro-F1: {grid.best_score_:.4f}\n")
y_pred = grid.predict(X_test)
print(classification_report(y_test, y_pred, digits=3))
print("Confusion matrix [ham, spam]:\n", confusion_matrix(y_test, y_pred))
Output
{'ham': 4825, 'spam': 747} Best params: {'clf__alpha': 0.1, 'tfidf__ngram_range': (1, 2), 'tfidf__sublinear_tf': True} Best CV macro-F1: 0.9688 precision recall f1-score support ham 0.984 1.000 0.992 1206 spam 1.000 0.893 0.944 187 accuracy 0.986 1393 macro avg 0.992 0.947 0.968 1393 weighted avg 0.986 0.986 0.985 1393 Confusion matrix [ham, spam]: [[1206 0] [ 20 167]]

Zero false alarms: not one real message was put in the spam folder. 20 of 187 spam messages got through (spam recall 0.893). For a spam filter that is usually the right trade-off — losing a real message is worse than seeing one spam. The grid search picked a small α (0.1), as expected for a dataset with thousands of messages.

Comparing the Variants with Cross-Validation

from sklearn.naive_bayes import BernoulliNB, ComplementNB
from sklearn.feature_extraction.text import CountVectorizer
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import cross_val_score
import time

models = {
    "MultinomialNB": Pipeline([("v", TfidfVectorizer(sublinear_tf=True)), ("c", MultinomialNB(alpha=0.1))]),
    "ComplementNB ": Pipeline([("v", TfidfVectorizer(sublinear_tf=True)), ("c", ComplementNB(alpha=0.1))]),
    "BernoulliNB  ": Pipeline([("v", CountVectorizer(binary=True)),       ("c", BernoulliNB(alpha=0.1))]),
    "LogisticReg  ": Pipeline([("v", TfidfVectorizer(sublinear_tf=True)), ("c", LogisticRegression(C=10, max_iter=1000))]),
}
for name, model in models.items():
    t0 = time.perf_counter()
    f1 = cross_val_score(model, X_train, y_train, cv=5, scoring="f1_macro")
    print(f"{name}: macro-F1 = {f1.mean():.3f} +/- {f1.std():.3f}   ({time.perf_counter()-t0:.1f}s)")
Output
MultinomialNB: macro-F1 = 0.966 +/- 0.013 (0.3s) ComplementNB : macro-F1 = 0.955 +/- 0.013 (0.3s) BernoulliNB : macro-F1 = 0.977 +/- 0.009 (0.3s) LogisticReg : macro-F1 = 0.965 +/- 0.009 (0.4s)

All four models are close (macro-F1 0.955–0.977), and each trains in well under a second. On these short messages BernoulliNB is the best — presence or absence of a word matters more than its count. ComplementNB is made for imbalanced data, yet here it is slightly behind. Always test the variants on your own data.

Section 10

When to Use Naïve Bayes

✅
Text Classification
Spam detection, news topics, document tagging, sentiment. MultinomialNB with TF-IDF is a strong, fast baseline and often close to much more complex models.
spam · topics · sentiment · intent
✅
Small Datasets
Naïve Bayes has few parameters (one probability per feature per class), so it needs much less data than deep learning or large tree ensembles.
low-data · cold start
✅
Speed and Updates
Training is just counting, and prediction is one sum per class — O(features × classes). New examples can be added with partial_fit without retraining from scratch.
streaming · on-device · online learning
✅
Many Classes
Handles many classes directly — each class gets its own prior and likelihoods. No one-vs-rest tricks needed. Good for language detection or product categories.
multi-class · hundreds of labels
❌
Strongly Correlated Features
Copies or near-copies of a feature are counted again and again, so probabilities become too extreme. Logistic Regression or tree models handle correlation much better.
try Logistic Regression
❌
Structured Tabular Data
On mixed numeric tables, Gaussian NB’s bell-curve and independence assumptions often break. Random Forest and Gradient Boosting usually win here.
try Random Forest
Section 11

Naïve Bayes vs Other Classifiers

Property Naïve Bayes Logistic Regression Random Forest SVM
Training speed ⚡ Very fast (one pass) Fast Moderate Slow on large N (kernel)
Prediction speed ⚡ Very fast Very fast Moderate Moderate
Text / NLP Excellent baseline Excellent Weak on sparse text Excellent (linear)
Probability calibration Often overconfident Usually good Moderate No probabilities by default
Missing values Possible in theory (skip the feature); scikit-learn needs imputation Needs imputation Supported in scikit-learn ≥ 1.4 Needs imputation
Feature scaling needed No Recommended No Yes
Correlated features Assumption violated Handles well Handles well Handles well
Data needed Very little Moderate Moderate–High Moderate
Section 12

Strengths vs Weaknesses

✅ Strengths
Very fast to train — one pass over the data
Very fast predictions
Strong baseline for text and NLP tasks
Scales to millions of sparse features
Works with small training sets
Multi-class out of the box
Easy to inspect — you can read the learned probabilities
No feature scaling needed
Can learn online with partial_fit
❌ Weaknesses
Independence assumption almost always false
Probabilities often too extreme (overconfident)
Zero frequency problem (needs smoothing)
Cannot model feature interactions
Gaussian NB struggles with non-bell-shaped features
Usually beaten by boosting on tabular data
Word order is ignored (bag-of-words)
Redundant features are double-counted

Try It — Double Counting

Why are Naïve Bayes probabilities so extreme? Add copies of the same clue and watch Naïve Bayes become more and more sure — even though no new information was added.

📢 Double CountingCopy the same piece of evidence again and again. Naïve Bayes counts every copy.
copies of the same word
P(word | spam)
P(word | ham)
📐
Fixing Overconfident Probabilities with Calibration

If you need reliable probabilities (risk scores, thresholds), wrap the model in scikit-learn’s CalibratedClassifierCV: method='isotonic' for large datasets or 'sigmoid' for small ones. For the class decision alone, the raw model is usually fine. 

import numpy as np
from sklearn.base import clone
from sklearn.calibration import CalibratedClassifierCV
from sklearn.metrics import brier_score_loss, log_loss

nb  = grid.best_estimator_                                   # the tuned NB pipeline
cal = CalibratedClassifierCV(clone(nb), method="isotonic", cv=5).fit(X_train, y_train)

y01 = (y_test == "spam").astype(int)
for name, model in [("raw NB    ", nb), ("calibrated", cal)]:
    p = model.predict_proba(X_test)[:, 1]
    print(f"{name}  Brier = {brier_score_loss(y01, p):.4f}   log loss = {log_loss(y01, p):.4f}")
Output
raw NB Brier = 0.0124 log loss = 0.0501 calibrated Brier = 0.0110 log loss = 0.0465

Both scores are “lower is better”. Calibration improves them a little (Brier 0.0124 → 0.0110). On this easy dataset the gain is small; on harder data with many correlated words it is often larger.

Section 13

Naïve Bayes in the Real World

Email Spam Filters
📧
  • Made popular by Paul Graham’s “A Plan for Spam” (2002)
  • SpamAssassin and other filters include a Bayes component
  • Learns from each user’s own “spam / not spam” clicks
  • Cheap enough to run on every incoming message
Medical Decision Support
🏥
  • Symptoms → likely conditions
  • Probabilities support (never replace) clinicians
  • Works with limited patient data
  • Easy to explain: each symptom’s evidence is visible
Sentiment & NLP
💬
  • Product review classification
  • Social media monitoring in real time
  • News category tagging at scale
  • Language detection from character n-grams
Section 14

Golden Rules

🎯 Naïve Bayes — Golden Rules
1
Always smooth. Never use α = 0 in production. A single zero probability silently removes a class: no error is shown, the answer is just wrong. The default alpha=1.0 is a safe start.
2
Match the variant to your features. Continuous numbers → GaussianNB. Word counts or TF-IDF → MultinomialNB. Yes/no features or short texts → BernoulliNB. Imbalanced text → try ComplementNB. Categories → CategoricalNB. Then compare them with cross-validation.
3
Use Naïve Bayes as a baseline. It trains in milliseconds and gives you a solid score to beat. If a complex model cannot beat it, check the complex model first.
4
Tune alpha with cross-validation — usually the most important setting. Try values on a log scale: 0.001, 0.01, 0.1, 0.5, 1, 2. Large datasets usually prefer small α; small or very sparse datasets need larger α.
5
Do not trust raw probabilities for high-stakes decisions. Use CalibratedClassifierCV when you need reliable probabilities — risk scores, fraud thresholds, medical triage. For a simple class decision, the raw output is fine.
6
Work in log-space when you code it yourself. Use log P(Class) + Σ log P(xᵢ | Class) — never multiply raw probabilities. Underflow is silent and gives wrong answers with no warning.
7
A violated independence assumption is normal. It mostly hurts the probabilities, not always the final class decision. That is why Naïve Bayes still works well on text, even though words in a sentence are clearly related.