Machine Learning 📂 Supervised Learning · 13 of 19 219 min read

SVM Kernels

Learn why linear classifiers break on complex data, how the kernel trick maps points to higher dimensions without doing the heavy math, and how to choose the right kernel (Linear, RBF, Polynomial, Sigmoid) in scikit-learn — with diagrams, interactive charts, and real Python examples.

🌐 Language
▶️
This Tutorial Has Seven Simulators

Look for the interactive boxes: Why a Straight Line Fails (Section 02), The Bouncer’s Platform (03), Four Kernels, Same Data (04), Strict vs Lenient Teacher: C (11), Torchlight vs Floodlight: Gamma (12), Live Grid Search (14) and Grid vs Random Search (15). They all run live in your browser.

Section 01

The Bouncer at the Nightclub

🎭 Real-World Story
Two Groups Who Refuse to Be Separated
A nightclub has two kinds of guests — VIPs with gold badges and regular guests with silver badges. On a quiet night the VIPs stand on the left of the dance floor and the regulars on the right. The bouncer stretches a straight rope across the middle — done, everyone is separated.

But on a busy night the VIPs stand in a circle in the middle of the floor, completely surrounded by regular guests. Now no straight rope works. However the bouncer turns it, some VIPs end up on the wrong side.

This is exactly the problem a linear SVM has. The kernel trick is the bouncer’s clever solution: change the floor into a bowl — the middle stays low and the edges rise. Now the VIPs and the regulars are at different heights, and a flat sheet (a plane) can separate them perfectly.

That change of height is the core idea behind SVM kernels. This tutorial goes step by step — why 2D fails, how a third dimension solves it, the four kernels you will use in Python, and then how to tune the two most important settings: C and γ (gamma).

💡
What Is a Kernel?

A kernel is a similarity function K(a, b). It gives the dot product of two points as if they had been moved into a higher-dimensional space — without ever doing that expensive move. It is a mathematical shortcut that makes high-dimensional geometry affordable.

Section 02

Why a Straight Line Fails

A basic (linear) SVM finds the widest possible straight line in 2D — or hyperplane in more dimensions — between two classes. This works beautifully when the data is linearly separable. Real data is often not that polite.

◯
Circles
Concentric rings
One class forms a ring around the other. No straight line can divide inside from outside.
🌀
Interleaved Spirals
Non-convex
Two classes spiral around each other. Only a very flexible, winding boundary can separate them.
🎯
Checkerboard
XOR pattern
Opposite corners belong to the same class. This is the classic XOR problem — the simplest pattern that no straight line can split.

Try It — Why a Straight Line Fails

Choose a data shape and train a linear SVM. Then switch to the RBF kernel and watch the boundary bend.

📏 Why a Straight Line FailsPick a data shape. Train a linear SVM, then an RBF SVM.
■ class +1   ● class −1   ━ SVM boundary   shading = predicted class
⚠️
The Linear SVM Assumption

A linear SVM assumes the classes can be split (almost) by a straight line or flat plane. Give it circular data and it still draws a line — with poor accuracy. The kernel trick removes this limit.

Section 03

Lifting Data into 3D — The Kernel Trick

💡 The Mental Model
From a Flat Table to a Bowl
Picture blue and red dots on a table (2D). The red dots form a small cluster in the middle; the blue dots form a ring around them. Now lift every dot to a height equal to its squared distance from the centre. The red dots near the centre stay low; the blue dots on the ring rise high.

From the side you now see the red dots at the bottom of a bowl and the blue dots high up on its sides. A flat sheet of cardboard — a hyperplane — can slide between them and separate them perfectly.

The new height was z = x² + y². That is a feature map φ(x) — a rule that creates a new dimension. A kernel is the shortcut that gives the dot products of those new points directly.

Try It — The Bouncer’s Platform

Press Lift into 3D, then move the plane height. When no point is on the wrong side, look at the top view: the flat plane has become a circle.

🎪 The Bouncer’s PlatformGive every point a height z = x² + y², then slide a flat plane up and down.
lift
plane height
rotate
■ VIPs (centre)   ● regular guests (ring)   dark outline = wrong side   ▱ cutting plane
the plane seen from above (2D)
The Slow Way
compute φ(a), φ(b), then φ(a) · φ(b)
Move both points into the high-dimensional space, then take the dot product. Expensive — and impossible when the space is infinite (RBF).
The Kernel Trick
K(a, b) = φ(a) · φ(b), computed directly
Get the same number straight from the original coordinates, without ever building φ(x). Fast.
🎯
Why the Trick Works

The SVM training and prediction formulas only need dot products between pairs of points, never the coordinates themselves. A kernel gives those dot products directly from the original data. You get the power of a 1,000-dimensional (or infinite-dimensional) space while doing arithmetic only in 2D. The price: the SVM must compare points in pairs, so training cost grows quickly with the number of rows.

Section 04

The Four Kernel Types

scikit-learn’s SVC has four built-in kernels, plus two ways to bring your own. Each kernel measures similarity in a different way — and therefore gives a different shape of decision boundary.

📏
Linear
A straight hyperplane, no transformation. Fastest. Best when there are many features (often more than samples) or the data is already linearly separable.
K(a,b) = aᵀb
🌐
RBF (Gaussian)
Radial Basis Function. Its feature space has infinitely many dimensions. The default in scikit-learn. Smooth, flexible boundaries; γ controls how local they are.
K(a,b) = exp(−γ‖a−b‖²)
📐
Polynomial
Includes feature interactions up to degree d. Useful when products of features (x₁·x₂) matter.
K(a,b) = (γaᵀb + r)ᵈ
🧠
Sigmoid
Inspired by neural-network activations. Rarely better than RBF, and not a valid kernel for every γ and r.
K(a,b) = tanh(γaᵀb + r)
🔧
Custom Kernel
Pass your own Python function to SVC. Useful for special similarity measures — strings, graphs, time series. It must return a valid (positive semi-definite) similarity matrix.
kernel=my_func
📊
Precomputed
Pass a ready-made similarity (Gram) matrix. Best when the similarity is expensive and you want to compute it once and reuse it.
kernel='precomputed'

Try It — Four Kernels, Same Data

All four kernels are trained live on the same data. Switch data shapes and compare. On circles, set the polynomial degree to 3 and untick coef0 r = 1 — then see what happens.

🧩 Four Kernels, Same DataEvery panel is an SVM with C = 3, trained live. Accuracy is on 500 new points.
poly degree d
Linear
RBF (γ = 1)
Polynomial
Sigmoid (γ = 0.5)
under each panel: test accuracy · number of support vectors (SV)
Section 05

Deep Dive — Each Kernel Type

① Linear Kernel

The simplest kernel: the plain dot product of two vectors. No transformation. The SVM finds a straight hyperplane.

Formula
K(a, b) = aᵀ · b
Just the dot product. The only hyperparameter is C.
Best For
High-dimensional data
Text classification, TF-IDF vectors, sparse data, many features.
from sklearn.svm import SVC
from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import make_pipeline

# 500 samples, 10 features, 5 of them informative
X, y = make_classification(n_samples=500, n_features=10, n_informative=5, random_state=42)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

model = make_pipeline(StandardScaler(), SVC(kernel="linear", C=1.0))
model.fit(X_train, y_train)
print(f"Linear kernel test accuracy: {model.score(X_test, y_test):.3f}")
print(f"Support vectors: {model[-1].n_support_.sum()} of {len(X_train)}")
Output
Linear kernel test accuracy: 0.810 Support vectors: 158 of 400

81% accuracy. This synthetic data is not fully linear, so a straight boundary leaves some errors — and 158 of 400 points end up as support vectors.

💡
Pro Tip — LinearSVC Is Faster

For large datasets with a linear kernel, use sklearn.svm.LinearSVC instead of SVC(kernel='linear'). It uses a different solver (liblinear instead of libsvm) whose training time grows roughly linearly with the number of rows, while SVC grows between n² and n³.

② RBF Kernel (Gaussian)

The Radial Basis Function kernel is the workhorse of SVMs. When you do not know which kernel to use, start here. Its feature space has infinitely many dimensions, and it creates smooth, flexible boundaries.

Formula
K(a, b) = exp(−γ ‖a − b‖²)
A bell curve of distance: 1 for identical points, close to 0 for far-apart points.
γ Controls the Reach
small γ → smooth · large γ → spiky
γ is like 1 / (influence radius)². Low γ = wide influence; high γ = narrow influence (overfits).
🎛️
γ and C — the Two Knobs You Always Tune

C sets the trade-off between a wide margin and few training mistakes. γ sets how far the influence of one training point reaches. Tune both together with GridSearchCV (Sections 11–14).

from sklearn.svm import SVC
from sklearn.datasets import make_circles
from sklearn.model_selection import train_test_split, GridSearchCV
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import Pipeline

# Circular data: a linear SVM would fail here
X, y = make_circles(n_samples=400, noise=0.1, factor=0.4, random_state=42)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

# Scaler INSIDE the pipeline, so each CV fold is scaled using only its own training part
pipe = Pipeline([("scaler", StandardScaler()), ("svm", SVC(kernel="rbf"))])
param_grid = {
    "svm__C":     [0.1, 1, 10, 100],
    "svm__gamma": ["scale", "auto", 0.01, 0.1, 1],
}
grid = GridSearchCV(pipe, param_grid, cv=5, scoring="accuracy")
grid.fit(X_train, y_train)
print(f"Best params:   {grid.best_params_}")
print(f"Best CV acc:   {grid.best_score_:.3f}")
print(f"Test accuracy: {grid.score(X_test, y_test):.3f}")
Output
Best params: {'svm__C': 0.1, 'svm__gamma': 'scale'} Best CV acc: 1.000 Test accuracy: 0.988

This circle dataset is easy for RBF: cross-validation reaches 100%. Notice that the StandardScaler sits inside the pipeline — if you scale all training data before the grid search, each validation fold has already “seen” the others (a small data leak).

③ Polynomial Kernel

The polynomial kernel adds feature interactions up to degree d. Degree 2 includes all pairwise products (x₁x₂, x₁², x₂²…). Degree 3 adds cubic terms.

🧮 What degree = 2 (with r = 1) Actually Creates
Input
Features [x₁, x₂] in 2D
Maps to
[1, x₁, x₂, x₁², x₁x₂, x₂²] (with some fixed weights) — 6 dimensions
With kernel
All these interactions are included without building the 6D vectors — just K(a, b) = (γaᵀb + r)²
Careful
With r = 0 (scikit-learn’s default coef0=0), only terms of exactly degree d remain. Degree 3 then has only cubic terms — no x₁², no x₂².
from sklearn.svm import SVC
from sklearn.datasets import make_moons
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import Pipeline
from sklearn.model_selection import cross_val_score

# Moon-shaped data: needs a curved boundary
X, y = make_moons(n_samples=500, noise=0.15, random_state=42)

pipe = Pipeline([
    ("scaler", StandardScaler()),
    ("svm", SVC(kernel="poly", degree=3, C=5, coef0=1, gamma="scale")),
])
scores = cross_val_score(pipe, X, y, cv=5, scoring="accuracy")
print(f"CV accuracy (degree=3): {scores.mean():.3f} +/- {scores.std():.3f}")

for d in [1, 2, 3, 4, 5]:
    pipe.set_params(svm__degree=d)
    s = cross_val_score(pipe, X, y, cv=5)
    print(f"  degree={d}: {s.mean():.3f}")
Output
CV accuracy (degree=3): 0.994 +/- 0.005 degree=1: 0.858 degree=2: 0.844 degree=3: 0.994 degree=4: 0.996 degree=5: 0.996

Degrees 1 and 2 are too simple for the moon shape (about 85%); degree 3 jumps to 99.4%. Degrees 4 and 5 add nothing more — so prefer the simpler degree 3.

④ Sigmoid Kernel

The sigmoid kernel behaves a little like a one-hidden-layer neural network. It is the least used of the four, and it is not a valid (positive semi-definite) kernel for all parameter values.

from sklearn.svm import SVC
from sklearn.datasets import make_classification
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import Pipeline
from sklearn.model_selection import cross_val_score

X, y = make_classification(n_samples=600, n_features=8, random_state=42)

for kernel in ["sigmoid", "rbf"]:
    pipe = Pipeline([("scaler", StandardScaler()),
                     ("svm", SVC(kernel=kernel, C=1.0, coef0=0, gamma="scale"))])
    scores = cross_val_score(pipe, X, y, cv=5)
    print(f"{kernel:8s} CV accuracy: {scores.mean():.3f}")
Output
sigmoid CV accuracy: 0.885 rbf CV accuracy: 0.913

Here the sigmoid kernel gets 88.5%, a little below RBF (91.3%). That is the usual story.

⚠️
Sigmoid Is Not Always a Valid Kernel

The sigmoid kernel satisfies Mercer’s condition only for some values of γ and r. Then the optimisation is no longer guaranteed to find the best answer. If sigmoid happens to be your best kernel, also try a real neural network.

Section 06

All Kernels — Side-by-Side Code

The clearest way to see the difference: train every kernel on the same dataset and compare.

from sklearn.svm import SVC
from sklearn.datasets import make_circles
from sklearn.model_selection import cross_val_score
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import Pipeline

# Circular data: the hardest case for a linear kernel
X, y = make_circles(n_samples=600, noise=0.1, factor=0.4, random_state=0)

kernels = {
    "linear":  SVC(kernel="linear",  C=1.0),
    "rbf":     SVC(kernel="rbf",     C=10,  gamma="scale"),
    "poly":    SVC(kernel="poly",    C=5,   degree=2, gamma="scale"),
    "poly d=3":SVC(kernel="poly",    C=5,   degree=3, gamma="scale"),
    "sigmoid": SVC(kernel="sigmoid", C=1.0, gamma="scale"),
}
print(f"{'Kernel':<10} {'Mean CV acc':>12} {'Std':>8}")
print("-" * 32)
for name, clf in kernels.items():
    pipe = Pipeline([("scaler", StandardScaler()), ("svm", clf)])
    scores = cross_val_score(pipe, X, y, cv=5, scoring="accuracy")
    print(f"{name:<10} {scores.mean():>12.3f} {scores.std():>8.3f}")
Output
Kernel Mean CV acc Std -------------------------------- linear 0.557 0.021 rbf 0.997 0.004 poly 0.995 0.007 poly d=3 0.632 0.028 sigmoid 0.575 0.029
📊
What the Numbers Tell You

On circles, linear (0.557) and sigmoid (0.575) are barely better than guessing (0.500). RBF (0.997) and polynomial degree 2 (0.995) solve the problem. The surprise: polynomial degree 3 gets only 0.632. With the default coef0=0 it has only cubic terms, and a circle needs x² + y². Kernel settings matter as much as the kernel name. This is also why RBF is the default first choice — it handles linear and curved data well without such traps.

Section 07

How to Choose the Right Kernel

01
Start with RBF
Begin with SVC(kernel='rbf') inside a pipeline with StandardScaler. Run a grid search over C and γ. If the cross-validated score is good enough for your problem and training is fast enough, you may be done.
02
Switch to linear if: very many features or text
Sparse, high-dimensional data (TF-IDF, one-hot text) is already in a huge space, and is often almost linearly separable. A kernel adds cost but little accuracy. Use LinearSVC for speed.
03
Try polynomial if: feature interactions matter
If you believe products of features (x₁ × x₂) carry information, try degree 2 or 3 — and remember to tune coef0 too.
04
Avoid sigmoid unless you have a reason
Sigmoid rarely beats RBF. If you want neural-network-like behaviour, use an actual neural network.
05
Scale your data — always
All kernels use distances or dot products, so features with large numbers dominate. Put StandardScaler() in the pipeline before the SVM.
KernelBest situationKey paramsScale data?Speed
LinearText, high-dimensional sparse data, features > samplesCRecommendedFast
RBFDefault choice, unknown structureC, gammaRequiredMedium
PolynomialFeature interactionsC, degree, coef0, gammaRequiredMedium
SigmoidSpecial cases onlyC, gamma, coef0RequiredMedium
CustomDomain-specific similarity (graphs, strings)Your functionDependsOften slow
Section 08

Complete Worked Example — Iris Dataset

Now everything together in one real pipeline: hold out a test set, scale, search over kernels and their own hyperparameters, then evaluate once on the test set.

from sklearn.datasets import load_iris
from sklearn.svm import SVC
from sklearn.model_selection import train_test_split, GridSearchCV
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import Pipeline
from sklearn.metrics import classification_report

# 1. Load data
iris = load_iris()
X, y = iris.data, iris.target

# 2. Hold out a true test set (never touched during tuning)
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42, stratify=y)

# 3. Pipeline: scaling + SVM
pipe = Pipeline([("scaler", StandardScaler()), ("svm", SVC())])

# 4. Search over kernels AND their own hyperparameters
param_grid = [
    {"svm__kernel": ["linear"], "svm__C": [0.1, 1, 10]},
    {"svm__kernel": ["rbf"],    "svm__C": [0.1, 1, 10, 100], "svm__gamma": ["scale", 0.01, 0.1]},
    {"svm__kernel": ["poly"],   "svm__C": [1, 10], "svm__degree": [2, 3], "svm__gamma": ["scale"]},
]
grid = GridSearchCV(pipe, param_grid, cv=5, scoring="accuracy", n_jobs=-1)
grid.fit(X_train, y_train)

# 5. Evaluate once on the held-out test set
best = grid.best_estimator_
print(f"Best params  : {grid.best_params_}")
print(f"CV accuracy  : {grid.best_score_:.3f}")
print(f"Test accuracy: {best.score(X_test, y_test):.3f}\n")
print(classification_report(y_test, best.predict(X_test), target_names=iris.target_names, digits=3))
Output
Best params : {'svm__C': 0.1, 'svm__kernel': 'linear'} CV accuracy : 0.975 Test accuracy: 0.933 precision recall f1-score support setosa 1.000 1.000 1.000 10 versicolor 0.900 0.900 0.900 10 virginica 0.900 0.900 0.900 10 accuracy 0.933 30 macro avg 0.933 0.933 0.933 30 weighted avg 0.933 0.933 0.933 30

The winner is the simplest model — a linear kernel with C = 0.1 (CV 97.5%). On the 30 test flowers it gets 93.3% (two mistakes). With so few test points one flower is worth 3.3%, so the small gap between CV and test score is normal noise, not a sign of a bad model. Notice we did not set probability=True: it is only needed for predict_proba, and it makes training much slower.

Section 09

Visualising Decision Boundaries in Python

A picture is worth a hundred accuracy scores. This function plots the boundary, the margin lines and the support vectors for any 2D dataset and any kernel.

import numpy as np
import matplotlib.pyplot as plt
from sklearn.svm import SVC
from sklearn.datasets import make_circles
from sklearn.preprocessing import StandardScaler

def plot_kernel_boundary(X, y, kernel, C=1.0, gamma="scale", degree=3, ax=None, title=""):
    """Plot the 2D decision boundary of an SVM with the given kernel."""
    X_scaled = StandardScaler().fit_transform(X)
    model = SVC(kernel=kernel, C=C, gamma=gamma, degree=degree).fit(X_scaled, y)

    # A fine grid of points covering the plot
    x_min, x_max = X_scaled[:, 0].min() - 0.5, X_scaled[:, 0].max() + 0.5
    y_min, y_max = X_scaled[:, 1].min() - 0.5, X_scaled[:, 1].max() + 0.5
    xx, yy = np.meshgrid(np.arange(x_min, x_max, 0.02), np.arange(y_min, y_max, 0.02))
    Z = model.decision_function(np.c_[xx.ravel(), yy.ravel()]).reshape(xx.shape)

    if ax is None:
        _, ax = plt.subplots(figsize=(6, 5))
    ax.contourf(xx, yy, Z > 0, alpha=0.3, cmap="RdBu")
    ax.contour(xx, yy, Z, levels=[-1, 0, 1], colors="k", linestyles=["--", "-", "--"])  # margin and boundary
    ax.scatter(X_scaled[:, 0], X_scaled[:, 1], c=y, cmap="RdBu", edgecolors="k", s=30)

    sv = model.support_vectors_           # circle the support vectors
    ax.scatter(sv[:, 0], sv[:, 1], s=150, linewidths=1.5, edgecolors="orange", facecolors="none")
    ax.set_title(f"{title}  train acc: {model.score(X_scaled, y):.2f}  SVs: {len(sv)}")
    return ax

X, y = make_circles(n_samples=300, noise=0.1, factor=0.4, random_state=42)
fig, axes = plt.subplots(1, 4, figsize=(20, 4))
for ax, kern in zip(axes, ["linear", "rbf", "poly", "sigmoid"]):
    plot_kernel_boundary(X, y, kernel=kern, C=10, ax=ax, title=kern.upper())
plt.tight_layout()
plt.savefig("kernel_comparison.png", dpi=150, bbox_inches="tight")
🟡
Reading the Boundary Plot

Orange circles mark the support vectors — the points on or inside the margin that define the boundary. The number of support vectors is a useful signal, but read it in both directions: very many support vectors can mean too small C (a very wide, soft margin), too small γ (a boundary that cannot fit), or too large γ (every point becomes its own little island). Section 12 shows all three in real numbers.

Section 10

Golden Rules — Kernels

🎯 SVM Kernels — Key Rules
1
Always scale your data before an SVM. Put StandardScaler inside a Pipeline to avoid data leakage.
2
Start with RBF. It is the most robust default. Switch to linear for very large or text data, or to polynomial when you know feature interactions matter.
3
Tune C and γ together with GridSearchCV or RandomizedSearchCV, on a log scale. A large C with a large γ overfits badly.
4
Kernel SVMs do not scale to very large datasets. Training cost grows between O(n²) and O(n³). For n > 100k, consider LinearSVC, SGDClassifier, a kernel approximation (Nystroem), or gradient boosting.
5
The kernel trick saves the mapping, not the pairs. You never build φ(x), but the SVM still compares points in pairs — the kernel matrix has n × n entries. That is why kernel SVMs get slow on big data.
6
Watch the support-vector count. If most training points are support vectors, check C and γ: the margin may be too soft (C too small) or γ may be too small or too large. Compare train and CV accuracy to see which.
Section 11

The C Parameter — Controlling the Margin

🎭 Real-World Story
The Strict vs Lenient Teacher
Two teachers mark the same homework. Teacher A (high C) is a perfectionist: she punishes every single mistake heavily, so her students learn to copy every detail of the homework — even the typos. They score very high on homework but can struggle on the real exam.

Teacher B (low C) is relaxed: she allows some mistakes and focuses on the big picture. If she is too relaxed, though, the students never learn enough.

C is that strictness dial. It controls how much the SVM is allowed to misclassify training points in exchange for a wider, safer margin.
Soft-Margin SVM Objective
minimise ½‖w‖² + C Σ ξᵢ
The margin width is 2/‖w‖, so a small ‖w‖ means a wide margin. ξᵢ (slack) is how far point i breaks the margin rule — 0 for points safely outside it.
The Trade-off
small C → wide margin, more errors
large C → narrow margin, fewer errors
C balances a wide margin against mistakes on the training data.

Try It — Strict vs Lenient Teacher

Move C from lenient to strict. Watch the boundary, the number of support vectors, and the curve of train vs test accuracy below.

👩‍🏫 Strict vs Lenient Teacher: CNoisy, overlapping rings. RBF kernel with γ = 1. Move C.
C (log scale)
◯ support vector   ━ boundary
accuracy for every value (computed live)
C valueMarginTraining accuracyGeneralisationUse when
0.001 – 0.1Very wideLowMay underfitVery noisy data
0.5 – 5BalancedMediumGoodDefault starting point — most problems
10 – 100NarrowHighWatch the gapClean data, little overlap
1000+Very thinVery highOverfit risk (especially with large γ)Only if cross-validation agrees
 
from sklearn.svm import SVC
from sklearn.datasets import make_classification
from sklearn.model_selection import cross_validate
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import Pipeline

X, y = make_classification(n_samples=500, n_features=2, n_redundant=0, n_informative=2,
                           n_clusters_per_class=1, flip_y=0.1, random_state=42)

print(f"{'C':<8} {'Train acc':>10} {'CV acc':>8} {'Gap':>7} {'# SVs':>6}")
print("-" * 43)
for c in [0.001, 0.01, 0.1, 1, 10, 100, 1000]:
    pipe = Pipeline([("sc", StandardScaler()), ("svm", SVC(kernel="rbf", C=c, gamma="scale"))])
    cv = cross_validate(pipe, X, y, cv=5, return_train_score=True)
    pipe.fit(X, y)
    tr, te = cv["train_score"].mean(), cv["test_score"].mean()
    print(f"{c:<8} {tr:>10.3f} {te:>8.3f} {tr-te:>7.3f} {pipe[-1].n_support_.sum():>6}")
Output
C Train acc CV acc Gap # SVs ------------------------------------------- 0.001 0.504 0.504 0.000 496 0.01 0.769 0.772 -0.003 496 0.1 0.854 0.844 0.010 259 1 0.887 0.886 0.001 179 10 0.900 0.890 0.010 145 100 0.913 0.900 0.013 129 1000 0.917 0.908 0.009 118
💡
The Train–CV Gap Is Your Overfitting Alarm

Read the table from top to bottom. At C = 0.001 the model just guesses (50%) and every point is a support vector. CV accuracy climbs fast up to C = 1 and then only slowly. Even at C = 1000 the gap stays below one percentage point here — with γ = 'scale' this 2D boundary stays smooth, so a strict C barely hurts. The lesson: always look at the gap. A gap of several points (as for large γ in the next section) is the red flag.

Section 12

The Gamma Parameter — Controlling the Reach

γ is used by the RBF, polynomial and sigmoid kernels. In the RBF kernel it sets how far the influence of one training point reaches — like the radius of vision of each support vector.

🔦 Mental Model
Torchlight vs Floodlight
Imagine every training point holds a lamp. High γ is a narrow torchlight: the point only “sees” its closest neighbours. The boundary wraps tightly around small groups of points, like cling film.

Low γ is a floodlight: each point lights up a wide area. The boundary becomes a smooth, gentle curve that sees the big picture.

Too narrow (high γ) and the model memorises every quirk of the training data. Too wide (low γ) and it misses the real shape of the classes.
RBF Formula
K(a,b) = exp(−γ ‖a−b‖²)
γ is in the exponent: a higher γ makes the bell curve narrower.
scikit-learn Options
'scale' = 1 / (n_features × X.var())
'auto' = 1 / n_features
'scale' (the default) adapts to the spread of your data.

Try It — Torchlight vs Floodlight

Move γ from floodlight to torchlight. The small chart shows how far one point’s influence reaches; the curve shows train and test accuracy for every γ.

🔦 Torchlight vs Floodlight: GammaNoisy, overlapping rings. RBF kernel with C = 3. Move γ.
γ (log scale)
◯ support vector   ━ boundary
accuracy for every value (computed live)
reach of one point: K = exp(−γ·d²)
γ (after scaling)Boundary shapeRiskWhen to try
0.001 – 0.01Very smooth, almost linearUnderfitsWhen RBF should behave almost like linear
'scale' (default)Adapted to the data spreadGood startAlways try this first
0.1 – 1Moderate curvesOften bestMost real datasets
1 – 10Tight around clustersWatch the gapClean, well-separated data
10 – 100+Wraps single pointsOverfit riskRarely — check CV carefully
 
from sklearn.svm import SVC
from sklearn.datasets import make_circles
from sklearn.model_selection import cross_validate, StratifiedKFold
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import Pipeline

X, y = make_circles(n_samples=400, noise=0.2, factor=0.5, random_state=42)   # noisy, overlapping rings
skf = StratifiedKFold(n_splits=5, shuffle=True, random_state=0)

print(f"{'gamma':<8} {'Train acc':>10} {'CV acc':>8} {'# SVs':>7}")
print("-" * 36)
for g in [0.001, 0.01, 0.1, 0.5, 1, 5, 50, 500]:
    pipe = Pipeline([("sc", StandardScaler()), ("svm", SVC(kernel="rbf", C=10, gamma=g))])
    cv = cross_validate(pipe, X, y, cv=skf, return_train_score=True)
    pipe.fit(X, y)
    n_sv = pipe.named_steps["svm"].support_vectors_.shape[0]
    print(f"{g:<8} {cv['train_score'].mean():>10.3f} {cv['test_score'].mean():>8.3f} {n_sv:>7d}")
Output
gamma Train acc CV acc # SVs ------------------------------------ 0.001 0.613 0.565 400 0.01 0.819 0.800 344 0.1 0.899 0.887 128 0.5 0.900 0.887 111 1 0.898 0.875 117 5 0.931 0.858 139 50 0.994 0.843 319 500 1.000 0.670 397
🔢
The Support-Vector Count Is U-Shaped

Look at the last column. At γ = 0.001 all 400 points are support vectors — the boundary is too smooth to fit, so almost every point is inside the margin. The count falls to about 111 at γ = 0.5, where CV accuracy is best (0.887). Then it rises again: at γ = 500 there are 397 support vectors, train accuracy is 100% but CV falls to 0.670 — every point has become its own island. So many support vectors can mean underfitting or overfitting; the train–CV gap tells you which.

Section 13

C and Gamma — How They Interact

C and γ do not act alone. Together they decide whether the model underfits, generalises well, or memorises the data. Tune them jointly — never one at a time.

Low γ (smooth)High γ (spiky)
Low C (soft, wide margin)Underfit — too simpleCan be OK — spiky shape, but mistakes are cheap
High C (strict)Often good — smooth boundary, few mistakesOverfit — tight, spiky, memorises the training data
⚠️
Never Tune C and γ Separately

A common mistake is to tune C with γ fixed, then tune γ with C fixed. This misses the interaction: good settings lie along a diagonal — a larger C usually needs a smaller γ. Always search both together in one 2D grid, as in the next section.

Section 14

Grid Search — Exhaustive Hyperparameter Tuning

Grid search tries every combination of the values you list and uses cross-validation to pick the best one.

🔍 How GridSearchCV Works
Step 1
Define a grid — for example C ∈ {0.1, 1, 10, 100} × γ ∈ {0.001, 0.01, 0.1, 1} = 16 combinations.
Step 2
For each combination, run k-fold cross-validation (e.g. cv = 5) and record the mean score. Total fits = 16 × 5 = 80.
Step 3
Pick the combination with the highest mean CV score, and refit it on the full training set.
Step 4
Evaluate on the held-out test set — once, after the search, never during it.

Try It — Live Grid Search

Press Run grid search and watch every cell being cross-validated. Then click cells to compare their decision boundaries.

🔍 Live Grid Search6 values of C × 5 values of γ, 3-fold cross-validation on 150 noisy points.
cell = mean CV accuracy (%)   brighter = better   ★ = best

Complete GridSearchCV Code

from sklearn.svm import SVC
from sklearn.datasets import make_circles
from sklearn.model_selection import GridSearchCV, train_test_split
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import Pipeline
import pandas as pd
import numpy as np

X, y = make_circles(n_samples=600, noise=0.2, factor=0.5, random_state=42)   # noisy, overlapping rings
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, stratify=y, random_state=42)

pipe = Pipeline([("sc", StandardScaler()), ("svm", SVC(kernel="rbf"))])

param_grid = {
    "svm__C":     [0.1, 1, 10, 100, 500],
    "svm__gamma": [0.001, 0.01, 0.1, 1, 10],
}   # 5 x 5 = 25 combinations x 5 folds = 125 fits

grid_search = GridSearchCV(pipe, param_grid, cv=5, scoring="accuracy",
                           n_jobs=-1, refit=True)   # refit the best model on all training data
grid_search.fit(X_train, y_train)

print("Best params :", grid_search.best_params_)
print(f"Best CV acc : {grid_search.best_score_:.4f}")
print(f"Test acc    : {grid_search.score(X_test, y_test):.4f}\n")

results = pd.DataFrame(grid_search.cv_results_)
top5 = results.sort_values("rank_test_score")[
    ["param_svm__C", "param_svm__gamma", "mean_test_score", "std_test_score"]].head(5)
print(top5.round(4).to_string(index=False))
Output
Best params : {'svm__C': 1, 'svm__gamma': 0.1} Best CV acc : 0.8771 Test acc : 0.9333 param_svm__C param_svm__gamma mean_test_score std_test_score 1.0 0.10 0.8771 0.0202 100.0 0.01 0.8771 0.0202 500.0 0.10 0.8771 0.0202 500.0 0.01 0.8729 0.0138 100.0 0.10 0.8729 0.0241
✅
Reading the Top-5 Table

Three combinations tie at 0.8771 — along the same diagonal (C = 1, γ = 0.1), (C = 100, γ = 0.01), (C = 500, γ = 0.1). When scores tie, prefer the simpler, smoother model (smaller C and γ) or the one with lower std_test_score. Here GridSearchCV simply took the first one, C = 1 and γ = 0.1, which is also the simplest.

Plotting the Grid Search Results

import matplotlib.pyplot as plt

# cv_results_ follows the order of the grid: C changes slowest, gamma fastest
scores = grid_search.cv_results_["mean_test_score"].reshape(
    len(param_grid["svm__C"]), len(param_grid["svm__gamma"]))
print(scores.round(2))

fig, ax = plt.subplots(figsize=(7, 5))
im = ax.imshow(scores, cmap="viridis", origin="lower")
ax.set_xticks(range(len(param_grid["svm__gamma"])), labels=param_grid["svm__gamma"])
ax.set_yticks(range(len(param_grid["svm__C"])), labels=param_grid["svm__C"])
ax.set_xlabel("gamma"); ax.set_ylabel("C")
for i in range(scores.shape[0]):
    for j in range(scores.shape[1]):
        ax.text(j, i, f"{scores[i, j]:.2f}", ha="center", va="center", color="white")
plt.colorbar(im, label="CV accuracy")
plt.tight_layout()
plt.savefig("gridsearch_heatmap.png", dpi=150)
Output
[[0.6 0.62 0.83 0.86 0.85] [0.6 0.62 0.88 0.86 0.86] [0.6 0.84 0.87 0.87 0.83] [0.62 0.88 0.87 0.85 0.8 ] [0.74 0.87 0.88 0.84 0.78]]

Rows are C (0.1 at the top of the printout, 500 at the bottom), columns are γ (0.001 → 10). You can see the diagonal ridge of good scores, the weak bottom-left corner of the plot (both small), and the drop in the top-right corner (both large).

⏲️
Grid Search Cost Grows Fast

With 4 values of C, 5 of γ, 2 kernel degrees and 5 folds you already need 200 fits. Every new parameter multiplies the cost. That is why random search is often better value with many parameters.

Section 15

Random Search — Smarter Hyperparameter Tuning

Random search samples combinations at random from distributions you choose, instead of testing every point of a grid. It sounds less careful, but it is often more efficient.

🎲 The Key Insight
Why Random Can Beat a Grid
Treasure is buried in a field. Grid search puts flags in a neat 5×5 grid — 25 spots, evenly spaced.

But suppose only one direction matters — the treasure lies in a narrow stripe. Your 5×5 grid tests only 5 different positions along that important direction; the other 20 flags just repeat them at different, unimportant positions.

Random search scatters 25 flags randomly, so it tests 25 different positions along every direction — including the important one. Bergstra & Bengio (2012) showed that random search usually finds equally good or better settings than grid search for the same budget, because usually only a few hyperparameters really matter.

Try It — Grid vs Random Search

A toy score landscape where only log C matters. Change the budget, draw new random points, and press Repeat 500 times to see how often random search wins.

🎲 Grid vs Random SearchA toy score landscape: only log C matters (the bright stripe); log γ hardly matters.
budget (number of trials)
Grid search
Random search
● trial   ★ best trial   bright stripe = good values of C

RandomizedSearchCV with Continuous Distributions

from sklearn.model_selection import RandomizedSearchCV
from scipy.stats import loguniform

pipe = Pipeline([("sc", StandardScaler()), ("svm", SVC())])
param_dist = {
    "svm__kernel": ["rbf", "poly", "linear"],
    "svm__C":      loguniform(1e-2, 1e4),   # every order of magnitude equally likely
    "svm__gamma":  loguniform(1e-4, 1e1),
    "svm__degree": [2, 3, 4],               # only used when kernel='poly'
}
rand_search = RandomizedSearchCV(pipe, param_dist, n_iter=60, cv=5,
                                 scoring="accuracy", n_jobs=-1, random_state=42)
rand_search.fit(X_train, y_train)

best = {k: (round(float(v), 4) if isinstance(v, float) else v) for k, v in rand_search.best_params_.items()}
print("Best params :", best)
print(f"Best CV acc : {rand_search.best_score_:.4f}")
print(f"Test acc    : {rand_search.score(X_test, y_test):.4f}")
Output
Best params : {'svm__C': 107.8483, 'svm__degree': 4, 'svm__gamma': 0.0071, 'svm__kernel': 'rbf'} Best CV acc : 0.8812 Test acc : 0.9167

60 random combinations across three kernels found RBF with C ≈ 108 and γ ≈ 0.007 — another point on the same diagonal ridge as the grid search (CV 0.8812). The degree value is ignored because the kernel is RBF.

💡
loguniform vs uniform for C and γ

Always use loguniform for C and γ. Good values can be anywhere from 0.01 to 10,000. With uniform(0.01, 10000), about 90% of the samples would be above 1,000. Log-uniform samples every order of magnitude equally often.

Grid Search vs Random Search — Direct Comparison

import time
from sklearn.model_selection import GridSearchCV, RandomizedSearchCV

pipe = Pipeline([("sc", StandardScaler()), ("svm", SVC(kernel="rbf"))])
C_range     = np.logspace(-2, 4, 7)     # 7 values
gamma_range = np.logspace(-4, 1, 6)     # 6 values -> 42 combinations

t0 = time.perf_counter()
gs = GridSearchCV(pipe, {"svm__C": C_range, "svm__gamma": gamma_range}, cv=5, n_jobs=-1)
gs.fit(X_train, y_train)
t_grid = time.perf_counter() - t0

t0 = time.perf_counter()
rs = RandomizedSearchCV(pipe, {"svm__C": loguniform(1e-2, 1e4), "svm__gamma": loguniform(1e-4, 10)},
                        n_iter=42, cv=5, n_jobs=-1, random_state=42)   # same budget: 42 combinations
rs.fit(X_train, y_train)
t_rand = time.perf_counter() - t0

print(f"Grid   -> CV: {gs.best_score_:.4f}  test: {gs.score(X_test, y_test):.4f}  time: {t_grid:.1f}s")
print(f"Random -> CV: {rs.best_score_:.4f}  test: {rs.score(X_test, y_test):.4f}  time: {t_rand:.1f}s")
Output
Grid -> CV: 0.8771 test: 0.9333 time: 3.1s Random -> CV: 0.8771 test: 0.9167 time: 3.0s

With only two parameters and the same budget (42 combinations), both methods reach the same CV score (0.8771) in about the same time. The different test scores (0.933 vs 0.917) are just 2 of 120 test points — noise. Random search pays off when there are more parameters and only a few of them matter.

Grid SearchRandom Search
CoverageEvery combination of the listed valuesRandom samples from distributions
Best for1–3 parameters, small ranges, fast models3+ parameters, wide ranges, continuous values
Compute costMultiplies with every new parameterFixed: n_iter × n_folds
ReproducibilityFully deterministicSet random_state
Finds the exact grid optimumYes (within the grid)No — but it can find good values between grid points
Recommended for SVMQuick search over C and γSearch over kernel + C + γ + degree
🏆 Best Practice — Two-Stage Search
Stage 1
Coarse random search — wide log-uniform ranges, n_iter = 30–50, cv = 3. Goal: find the right order of magnitude for C and γ.
Stage 2
Fine grid search — a small grid around the Stage 1 winner (×0.1 to ×10), cv = 5. Goal: polish the answer.
Evaluate
Test the final model once on the held-out test set. Never look at the test set during either stage.
# Stage 1: coarse random search over a wide range
coarse = RandomizedSearchCV(
    pipe,
    {"svm__C": loguniform(1e-3, 1e5), "svm__gamma": loguniform(1e-5, 1e2)},
    n_iter=40, cv=3, n_jobs=-1, random_state=42)
coarse.fit(X_train, y_train)
best_C, best_gamma = coarse.best_params_["svm__C"], coarse.best_params_["svm__gamma"]
print(f"Stage 1 best: C={best_C:.2f}  gamma={best_gamma:.4f}  CV={coarse.best_score_:.4f}")

# Stage 2: fine grid around the Stage 1 result (x0.1 ... x10)
fine_C     = np.logspace(np.log10(best_C / 10),     np.log10(best_C * 10),     5)
fine_gamma = np.logspace(np.log10(best_gamma / 10), np.log10(best_gamma * 10), 5)
fine = GridSearchCV(pipe, {"svm__C": fine_C, "svm__gamma": fine_gamma},
                    cv=5, n_jobs=-1)
fine.fit(X_train, y_train)
p = fine.best_params_
print(f"Stage 2 best: C={p['svm__C']:.2f}  gamma={p['svm__gamma']:.4f}  CV={fine.best_score_:.4f}")
print(f"Final test accuracy: {fine.score(X_test, y_test):.4f}")
Output
Stage 1 best: C=717.71 gamma=0.1551 CV=0.8750 Stage 2 best: C=71.77 gamma=0.0155 CV=0.8812 Final test accuracy: 0.9250
🏆
Why Two Stages Are Efficient

Stage 1 used 40 × 3 = 120 fits to find the right region (C ≈ 700, γ ≈ 0.16). Stage 2 used 25 × 5 = 125 fits and moved along the ridge to C ≈ 72, γ ≈ 0.016 (CV 0.8812). A single fine grid over the whole range (C from 0.001 to 100,000 and γ from 0.00001 to 100) at the same resolution would need thousands of fits.

Section 16

Hyperparameter Tuning — Golden Rules

🎯 C, Gamma and Search — Key Rules
1
Scale first, always. Good values of C and γ depend on the scale of your features. Put StandardScaler inside a Pipeline so it is refitted inside every CV fold (no data leakage).
2
Start with gamma='scale'. It is the default and adapts to the spread of your data. Search around it with explicit values only when needed.
3
Search C and γ together, on a log scale. Use np.logspace(-3, 4, 8), not range(1, 100). These parameters work across many orders of magnitude, so linear spacing wastes most of the budget.
4
Use random search for 3 or more parameters. A grid with 4 parameters × 5 values = 625 combinations. Random search with n_iter = 50 tests 50 different values of every parameter.
5
Watch the train–CV gap — your overfitting detector. If train accuracy is 0.99 but CV is 0.87, you are overfitting: reduce γ, reduce C, or both.
6
Hold out a true test set and use it only once. Every time you check the test score and change a parameter, you are fitting to the test set. Use cross-validation during the search; report the test score once at the end.