Machine Learning 📂 Supervised Learning · 12 of 19 198 min read

Support Vector Machine

Learn how SVM finds the optimal decision boundary by maximising the margin between classes, handles non-linear data with the kernel trick, and delivers exceptional performance in high-dimensional spaces like text and image classification.

🌐 Language
▶️
This Tutorial Has Six Simulators

Look for the interactive boxes: Margin Explorer (Section 01), Soft Margin and C (05), The Kernel Trick (06), Kernel Playground (07), Scaling Matters (08) and SVR: the Epsilon Tube (09). Every SVM in them is trained live in your browser.

Section 01

The Story That Explains SVM

📖 Real-World Analogy
The Border Guard Who Wanted the Widest No-Man’s-Land
Two kingdoms — Kingdom Red and Kingdom Blue — share a disputed territory. A new border guard must draw the official border between them. Many lines could separate the two kingdoms without touching any village. But the guard is wise.

He does not draw just any line. He says: “I want the line that keeps the nearest village on each side as far from the border as possible — a wide, safe no-man’s-land on both sides.” He looks only at the villages nearest to the border, because they decide where the line must go. He calls them his support villages. Every other village has no effect on the border.

The wider the no-man’s-land, the more confident he is that future travellers will be put in the right kingdom, even if their position is a little uncertain. This widest no-man’s-land is the margin. The border line is the hyperplane. The support villages are the support vectors.

That is a Support Vector Machine: find the boundary with the widest gap between the two classes.
🔑
The Core Idea in One Sentence

SVM finds the hyperplane with the largest margin — the largest distance between the decision boundary and the nearest points of each class. A wide margin usually means better results on new, unseen data.

Try It — Margin Explorer

Drag the points and watch the street (the margin) change. Tick draw my own line and turn it: you will never beat the SVM street. Then press Keep only support vectors.

🛣️ Margin ExplorerDrag any point. Click an empty spot to add one. SVM always finds the widest street.
my line angle
━ hyperplane w·x+b=0   - - - margin lines w·x+b=±1   ◯ support vector   ━ my line
Section 02

The Hyperplane — What SVM Is Drawing

In 2D a decision boundary is a line. In 3D it is a plane. In more dimensions it is called a hyperplane — the same idea, just harder to picture. The SVM maths is the same in any number of dimensions.

Dimensions Number of Features Decision Boundary Example
2D 2 features A straight line Height vs weight → classify sport
3D 3 features A flat plane Height, weight, age → classify disease
nD n features A hyperplane 10,000 word features → classify spam
Hyperplane Equation
w · x + b = 0
w = weight vector (perpendicular to the hyperplane), x = feature vector, b = bias (shifts the boundary)
Classification Rule
ŷ = sign(w · x + b)
If w·x + b > 0 → class +1. If w·x + b < 0 → class −1. The sign decides the class.
📐 Hyperplane, Margin and Support Vectors
● ● ●
Class +1 points sit on one side. The closest ones lie on the upper margin line: w · x + b = +1
— — —
The hyperplane (decision boundary): w · x + b = 0 — exactly halfway between the two margin lines. A point on the other side of it gets the other class.
○ ○ ○
Class −1 points sit on the other side. The closest ones lie on the lower margin line: w · x + b = −1
Margin
The gap between the two margin lines = 2 / ‖w‖. So SVM makes the margin as wide as possible by making ‖w‖ as small as possible. Only the support vectors (the points on the margin lines) decide where the boundary goes.
Section 03

Support Vectors — The Only Points That Matter

📖 Key Insight
Delete Most of the Data — the Model Does Not Change
Suppose you train an SVM on 10,000 points to classify tumours as malignant or benign, and it ends up with 400 support vectors. After training, you could delete the other 9,600 points — the ones that are safely outside the margin — and train again on only the support vectors. You would get the same boundary.

This is the defining property of SVM: the boundary depends only on the points closest to it (and, with a soft margin, on the points inside the margin). Points far from the boundary have no effect. That is why SVM is not disturbed by outliers that are far away on the correct side, and why a trained SVM only needs to store its support vectors to make predictions.

You can test this yourself in the Margin Explorer in Section 01: press Keep only support vectors. The other points disappear, the SVM is trained again, and the boundary stays exactly where it was.

📌
Inspecting Support Vectors in scikit-learn

After fitting, use clf.support_vectors_ for the support vectors, clf.support_ for their row numbers, and clf.n_support_ for the count per class. If almost every training point is a support vector, the margin is very wide (small C), the data overlaps a lot, or the model is too simple. Few support vectors usually means clean, well-separated data.

Section 04

The SVM Objective — What Is Being Optimised

Training an SVM is an optimisation problem. It looks for the weight vector w and bias b that give the widest margin while classifying all training points correctly.

Primal Objective (Minimise)
½ ‖w‖²
Making ‖w‖ small makes the margin 2/‖w‖ large. We minimise ½‖w‖² because it is smooth and convex — easy to optimise, with one global best answer.
Subject To (Hard Margin)
yᵢ (w · xᵢ + b) ≥ 1   ∀i
Every training point must be on the correct side of its margin line. Because yᵢ is +1 or −1, one rule covers both classes.
⚠️
The Hard Margin Fails on Real Data

The hard margin needs every point on the correct side with no exceptions. If the classes overlap, or just one point is mislabelled, there is no solution at all. Real data is almost never perfectly separable. The answer is the soft margin.

Section 05

Hard Margin vs Soft Margin — The C Parameter

The soft-margin SVM adds slack variables ξᵢ. They let some points break the margin rule, but each violation has a cost. The hyperparameter C sets that cost.

Soft Margin Objective
min ½‖w‖² + C Σ ξᵢ
A small ‖w‖ (wide margin) plus a small total violation. C sets the balance between these two goals.
Soft Margin Constraint
yᵢ(w · xᵢ + b) ≥ 1 − ξᵢ,   ξᵢ ≥ 0
ξᵢ = 0: outside the margin, no cost. 0 < ξᵢ ≤ 1: inside the margin but still correct. ξᵢ > 1: on the wrong side (misclassified).
The Same Thing as a Loss Function — Hinge Loss
min ½‖w‖² + C Σ max(0, 1 − yᵢ(w · xᵢ + b))
The slack ξᵢ is exactly the hinge loss of point i. Points safely outside the margin have zero loss, so they do not affect the answer.
C Value Margin Width Violations Allowed Bias–Variance Risk
Very small (0.001) Very wide Many — very tolerant High bias (underfitting) Too simple, misclassifies many
Small (0.1) Wide Some violations OK Leans towards underfitting Smoother, more general
Medium (1.0) Balanced Moderate Balanced ✅ Good default starting point
Large (100) Narrow Rarely allowed High variance (overfitting) Can memorise training noise
∞ (Hard Margin) As narrow as needed Zero tolerance Extreme variance No solution on non-separable data
🎯
How to Think About C

C is how much you care about mistakes on the training data. High C = “I hate mistakes — fit the training data tightly.” Low C = “A few mistakes are fine — give me a wide, simple boundary.” Always tune C with cross-validation. Start at 1.0 and search on a log scale: 0.001 → 0.01 → 0.1 → 1 → 10 → 100.

Try It — Soft Margin and C

The two classes overlap, so a hard margin is impossible. Slide C from small to large and watch the street get narrower, the number of support vectors fall, and the test accuracy change.

🎚️ Soft Margin and C50 training points that overlap. Move C and watch the street and the support vectors.
C (log scale)
◯ support vector   ✕ wrong side of the boundary   shaded = the street (margin)
Section 06

The Kernel Trick — Handling Non-Linear Data

📖 Story
Lifting the Paper
Two groups of points are drawn on a sheet of paper: red points in the centre, blue points in a ring around them. No straight line can separate them in 2D. Now imagine pushing the centre of the paper down and lifting the edges up, like a bowl. The red points sit at the bottom of the bowl and the blue points sit high on its sides. A flat plane can now cut between them perfectly.

The kernel trick does exactly this. It maps the data into a higher-dimensional space where a flat hyperplane can separate it. The clever part: the SVM never computes the new coordinates. It only needs dot products between points, and a kernel function K(xᵢ, xⱼ) gives those directly. Fast, exact and elegant.
01
Original Feature Space (not linearly separable)
Data in 2D — a blue ring around a red cluster. No straight line can separate them. A linear SVM fails.
02
Apply a Kernel — Implicit Mapping φ(x)
The kernel K(xᵢ, xⱼ) = φ(xᵢ) · φ(xⱼ) gives dot products in the new space without ever computing φ(x). For the RBF kernel the new space even has infinitely many dimensions — yet K costs almost nothing to compute.
03
Higher-Dimensional Space (linearly separable)
In the new space the data separates cleanly. SVM finds the maximum-margin hyperplane there, with the same algorithm as before.
04
Back in the Original Space
The flat hyperplane in the high-dimensional space looks like a curved boundary in the original space — a circle, an ellipse, or a more complex shape, depending on the kernel.

Try It — The Kernel Trick

Press Lift into 3D. Each point gets a height z = x² + y². Watch a flat plane separate the classes, and see what that plane looks like back in 2D.

🪄 The Kernel TrickA blue ring around a red centre. No straight line separates them in 2D.
lift z = x² + y²
rotate the view
■ class +1 (centre)   ● class −1 (ring)   ▱ flat plane in 3D
back in 2D
Section 07

The Four Kernels — Choosing the Right One

📏
Linear Kernel
K(xᵢ, xⱼ) = xᵢ · xⱼ
No transformation — the normal dot product. The boundary is a straight line or flat plane. Fastest to train. Works very well for high-dimensional data such as TF-IDF text vectors, where the feature space is already huge.
✅ Fast, readable weights, great for text
❌ Cannot draw curved boundaries
🔵
RBF Kernel
K(xᵢ, xⱼ) = exp(−γ‖xᵢ − xⱼ‖²)
Radial Basis Function — the most popular kernel. It measures similarity as a bell curve of distance: nearby points are very similar, distant points are not similar at all. It can make smooth curved boundaries of almost any shape. It is the default kernel in scikit-learn and a strong first choice.
✅ Works on most non-linear problems
❌ Two hyperparameters to tune (C and γ)
🌊
Polynomial Kernel
K(xᵢ, xⱼ) = (γ xᵢ · xⱼ + r)ᵈ
Raises the dot product to degree d. Degree 2 gives quadratic boundaries (parabolas, ellipses); degree 3 gives cubic shapes. Useful when interactions between features up to degree d matter.
✅ Models feature interactions directly
❌ Many hyperparameters (C, d, γ, r); can be unstable for high d
〰️
Sigmoid Kernel
K(xᵢ, xⱼ) = tanh(γ xᵢ · xⱼ + r)
Inspired by neural network activations. It is not a valid kernel for all values of γ and r, so training can behave badly. Kept mostly for historical reasons.
✅ Sometimes works on specific data
❌ Rarely better than RBF or linear
Kernel Key Parameters Effect of High Value Effect of Low Value Best For
Linear C only High C → tighter fit to the training data Low C → wider margin, more regularised Text, sparse high-dimensional data
RBF C and γ (gamma) High γ → wiggly boundary, overfits Low γ → smooth boundary, can underfit General purpose, most datasets
Polynomial C, degree d, γ, r High d → very complex boundary d = 1 → essentially linear Problems with feature interactions
Sigmoid C, γ, r Unstable, unpredictable behaviour Behaves close to linear Rarely used — prefer RBF or linear
⚠️
γ (gamma) — the Most Dangerous Hyperparameter

In the RBF kernel, γ controls how far each training point’s influence reaches. High γ → each point only affects its close neighbours → a jagged boundary around single points → overfitting. Low γ → each point affects a large area → a very smooth boundary → underfitting. In scikit-learn, gamma='scale' (the default) sets γ = 1 / (n_features × X.var()) — it adapts to the spread of your data. It is usually a better start than gamma='auto' (γ = 1 / n_features).

Try It — Kernel Playground

Pick a dataset and a kernel. Move C and γ and watch the boundary, the support vectors, and the gap between train and test accuracy. Click on the chart to add your own points.

🎛️ Kernel Playground100 training points. Pick data and a kernel, then tune C and γ. Click to add points.
C (log scale)
γ (log scale)
━ decision boundary f(x)=0   - - - f(x)=±1   ◯ support vector   colour = f(x)
Section 08

Python Implementation

SVC on the Iris Dataset — All Four Kernels

from sklearn.svm import SVC
from sklearn.datasets import load_iris
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import Pipeline

iris = load_iris()
X, y = iris.data, iris.target
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42, stratify=y)

# Always scale features before an SVM: it works with distances
for kernel in ["linear", "rbf", "poly", "sigmoid"]:
    pipe = Pipeline([
        ("scaler", StandardScaler()),
        ("svc", SVC(kernel=kernel, C=1.0, gamma="scale")),  # gamma = 1 / (n_features * X.var())
    ])
    pipe.fit(X_train, y_train)
    acc = pipe.score(X_test, y_test)
    nsv = pipe["svc"].n_support_.sum()           # total number of support vectors
    print(f"Kernel: {kernel:8s} | Test accuracy: {acc:.4f} | Support vectors: {nsv}")
Output
Kernel: linear | Test accuracy: 1.0000 | Support vectors: 23 Kernel: rbf | Test accuracy: 0.9667 | Support vectors: 47 Kernel: poly | Test accuracy: 0.9000 | Support vectors: 46 Kernel: sigmoid | Test accuracy: 0.9000 | Support vectors: 45

On this small test set (30 flowers) the linear kernel gets every flower right, and RBF misses one. With only 30 test points, one flower is 3.3% — so do not choose a kernel from one split. Use cross-validation, as in the grid search below.

Inspecting the Fitted Model

pipe = Pipeline([
    ("scaler", StandardScaler()),
    ("svc", SVC(kernel="rbf", C=1.0, gamma="scale", probability=True, random_state=42)),
])
pipe.fit(X_train, y_train)
svc = pipe["svc"]

print("Support vectors per class:", svc.n_support_)
print("Total support vectors:    ", svc.n_support_.sum(), "of", len(X_train), "training points")
print("Support vector shape:     ", svc.support_vectors_.shape)

# predict_proba needs probability=True when the model is created
y_prob = pipe.predict_proba(X_test)
y_pred = pipe.predict(X_test)
print("\nSample predictions:")
for i in range(4):
    print(f"  Predicted: {iris.target_names[y_pred[i]]:12s} ({y_prob[i].max()*100:.1f}% probability)")
Output
Support vectors per class: [10 18 19] Total support vectors: 47 of 120 training points Support vector shape: (47, 4) Sample predictions: Predicted: setosa (96.1% probability) Predicted: virginica (69.5% probability) Predicted: versicolor (93.6% probability) Predicted: versicolor (95.1% probability)

47 of the 120 training flowers are support vectors — mostly versicolor and virginica, the two species that overlap. Setosa is easy to separate, so it needs only 10. Note: predict_proba uses an extra calibration step (Platt scaling with internal cross-validation), so its most likely class can occasionally disagree with predict.

Hyperparameter Tuning — GridSearchCV

from sklearn.model_selection import GridSearchCV
import pandas as pd

pipe = Pipeline([("scaler", StandardScaler()), ("svc", SVC())])
param_grid = [
    {"svc__kernel": ["rbf"],    "svc__C": [0.01, 0.1, 1, 10, 100],
     "svc__gamma": ["scale", 0.001, 0.01, 0.1, 1]},
    {"svc__kernel": ["linear"], "svc__C": [0.01, 0.1, 1, 10, 100]},   # linear has no gamma
]
grid = GridSearchCV(pipe, param_grid, cv=5, scoring="accuracy", n_jobs=-1)
grid.fit(X_train, y_train)

print("Best parameters: ", grid.best_params_)
print(f"Best CV accuracy: {grid.best_score_:.4f}")
print(f"Test accuracy:    {grid.score(X_test, y_test):.4f}\n")

results = pd.DataFrame(grid.cv_results_)
top5 = results.nsmallest(5, "rank_test_score")[
    ["param_svc__kernel", "param_svc__C", "param_svc__gamma", "mean_test_score"]]
print(top5.to_string(index=False))
Output
Best parameters: {'svc__C': 1, 'svc__gamma': 0.1, 'svc__kernel': 'rbf'} Best CV accuracy: 0.9750 Test accuracy: 0.9667 param_svc__kernel param_svc__C param_svc__gamma mean_test_score rbf 1.0 0.1 0.975 rbf 10.0 0.01 0.975 rbf 100.0 0.001 0.975 linear 0.1 NaN 0.975 linear 1.0 NaN 0.975

Notice that many settings tie at 97.5%. Also notice the pattern: a larger C paired with a smaller γ gives the same score — C and γ work together, so always tune them together. The linear kernel has no γ, so it gets its own small grid.

LinearSVC — Fast for Large Datasets and Text

import pandas as pd
from sklearn.svm import LinearSVC
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.model_selection import train_test_split
from sklearn.pipeline import Pipeline
from sklearn.metrics import classification_report

# 20 Newsgroups (training part), 4 topics
url = "https://raw.githubusercontent.com/selva86/datasets/master/newsgroups.json"
news = pd.read_json(url)
topics = ["sci.space", "comp.graphics", "rec.sport.hockey", "talk.politics.guns"]
news = news[news["target_names"].isin(topics)]
X_train, X_test, y_train, y_test = train_test_split(
    news["content"], news["target_names"], test_size=0.25, random_state=42,
    stratify=news["target_names"])

text_pipeline = Pipeline([
    ("tfidf", TfidfVectorizer(sublinear_tf=True, ngram_range=(1, 2),
                              max_features=50000, stop_words="english")),
    ("clf", LinearSVC(C=1.0)),       # liblinear: fast on sparse, high-dimensional data
])
text_pipeline.fit(X_train, y_train)
y_pred = text_pipeline.predict(X_test)

print(f"Test accuracy: {text_pipeline.score(X_test, y_test):.4f}\n")
print(classification_report(y_test, y_pred, digits=3))
Output
Test accuracy: 0.9828 precision recall f1-score support comp.graphics 0.960 0.986 0.973 146 rec.sport.hockey 0.993 0.987 0.990 150 sci.space 0.993 0.959 0.976 148 talk.politics.guns 0.986 1.000 0.993 137 accuracy 0.983 581 macro avg 0.983 0.983 0.983 581 weighted avg 0.983 0.983 0.983 581

98.3% accuracy on four newsgroups, using up to 50,000 word and word-pair features. The linear kernel is enough here: in such a high-dimensional space, the topics are already almost linearly separable.

⚡
SVC vs LinearSVC — When to Use Each

Use LinearSVC when the data is large (more than about 10,000 rows), high-dimensional and sparse (text), or when you need fast training. It uses the liblinear library, which scales roughly linearly with the number of rows. Use SVC(kernel='linear') for small data, or when you need predict_proba() or the support vectors. For curved boundaries you need a kernel, so use SVC(kernel='rbf') — or an approximation such as Nystroem + LinearSVC for large data.

Why Scaling Is Not Optional

from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import cross_val_score
from sklearn.svm import SVC

X, y = load_breast_cancer(return_X_y=True)
print(f"Feature ranges: {X.min(axis=0).min():.3f} ... {X.max(axis=0).max():.0f}")

raw    = SVC(kernel="rbf")
scaled = Pipeline([("scaler", StandardScaler()), ("svc", SVC(kernel="rbf"))])
for name, model in [("Without scaling", raw), ("With StandardScaler", scaled)]:
    acc = cross_val_score(model, X, y, cv=5).mean()
    print(f"{name:20s}: CV accuracy = {acc:.4f}")
Output
Feature ranges: 0.000 ... 4254 Without scaling : CV accuracy = 0.9122 With StandardScaler : CV accuracy = 0.9736

The same SVM, the same data: 91.2% without scaling and 97.4% with it. The breast cancer features range from about 0.001 to 4,254, so without scaling the largest features control every distance.

Try It — Scaling Matters

Risk depends on both income and age. Untick and tick StandardScaler first and watch the boundary turn.

📏 Scaling MattersRisk depends on income AND age. RBF SVM with gamma=“scale”, C = 1.
■ risky   ● safe   ━ SVM boundary
Section 09

SVM for Regression — SVR

SVM can also do regression. Instead of a street that separates classes, SVR (Support Vector Regression) fits a tube of width 2ε around the prediction, while keeping the function as flat as possible. Points inside the tube cost nothing. Only points outside the tube are penalised.

SVR Objective
min ½‖w‖² + C Σ (ξᵢ + ξᵢ*)
Keep the model simple, plus the total error outside the ε-tube. ξᵢ and ξᵢ* are the slacks for points above and below the tube.
The ε-Insensitive Loss
L(y, ŷ) = max(0, |y − ŷ| − ε)
Zero loss when the prediction is within ε of the true value. A linear penalty outside the tube.
📐 SVR — The ε-Tube
Tube top
f(x) + ε — the upper edge of the no-penalty zone
Centre
f(x) = w · x + b — the prediction (kept as flat as possible)
Tube bottom
f(x) − ε — the lower edge. Points inside → zero penalty.
Outside
Points beyond ±ε are penalised by their distance outside the tube. Points on or outside the tube become the support vectors.

Try It — SVR: the Epsilon Tube

Make the tube wider and watch the support vectors disappear. Make it very narrow and almost every point becomes a support vector.

🚇 SVR: the Epsilon Tube45 noisy points with one outlier. Points inside the tube cost nothing.
ε (tube half-width)
C (log scale)
γ (log scale)
━ prediction f(x)   shaded = tube f(x) ± ε   ● inside   ● outside   ◯ support vector
ε-insensitive loss

Now a real dataset — California housing, 16,512 training rows:

import pandas as pd
from sklearn.svm import SVR
from sklearn.impute import SimpleImputer
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import Pipeline
from sklearn.metrics import r2_score, mean_absolute_error

# California housing (1990 census), 20,640 districts
url = "https://raw.githubusercontent.com/ageron/handson-ml2/master/datasets/housing/housing.csv"
df = pd.read_csv(url)
X = df.drop(columns=["median_house_value", "ocean_proximity"])
y = df["median_house_value"] / 100_000            # target in units of $100,000

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

svr_pipe = Pipeline([
    ("impute", SimpleImputer(strategy="median")),  # SVR cannot handle missing values
    ("scaler", StandardScaler()),                  # SVR always needs scaling
    ("svr", SVR(kernel="rbf", C=10, epsilon=0.1, gamma="scale")),
])
svr_pipe.fit(X_train, y_train)
y_pred = svr_pipe.predict(X_test)

print(f"R² score:        {r2_score(y_test, y_pred):.4f}")
print(f"MAE:             ${mean_absolute_error(y_test, y_pred) * 100_000:,.0f}")
print(f"Support vectors: {len(svr_pipe['svr'].support_):,} of {len(X_train):,} training rows")
Output
R² score: 0.7577 MAE: $37,287 Support vectors: 12,512 of 16,512 training rows

R² = 0.758, with a typical error of about $37,000. But look at the last line: 12,512 of 16,512 rows are support vectors. The target is in units of $100,000, so ε = 0.1 means a tube of only ±$10,000 — most houses fall outside it. A larger ε would give fewer support vectors and faster predictions. This fit also took minutes, while a Random Forest on the same data takes seconds.

📌
SVR Hyperparameters — What Controls What

ε (epsilon): half-width of the no-penalty tube, in the same units as y. Larger ε → fewer support vectors, smoother fit, less sensitive to noise. Smaller ε → more support vectors, tighter fit. C: penalty for points outside the tube. High C → the curve chases every point → overfitting risk. kernel and γ: the same as for classification.

Section 10

Feature Importance with a Linear SVM

A linear SVM has a weight vector w with one number per feature. After scaling, the size of each weight shows how strongly that feature pushes a prediction towards one class, and its sign shows which class. 

import pandas as pd
from sklearn.svm import LinearSVC
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import Pipeline
from sklearn.datasets import load_breast_cancer

cancer = load_breast_cancer()
X, y = cancer.data, cancer.target
print("Class names:", dict(enumerate(cancer.target_names)))   # 0 = malignant, 1 = benign

pipe = Pipeline([
    ("scaler", StandardScaler()),               # coefficients are comparable only after scaling
    ("svm", LinearSVC(C=1.0, max_iter=10000)),
])
pipe.fit(X, y)

# Positive coefficient -> pushes towards class 1 (BENIGN)
# Negative coefficient -> pushes towards class 0 (MALIGNANT)
coef = pd.Series(pipe["svm"].coef_[0], index=cancer.feature_names)
print("\nTop features pushing towards MALIGNANT (most negative):")
print(coef.sort_values().head(5).round(3).to_string())
print("\nTop features pushing towards BENIGN (most positive):")
print(coef.sort_values(ascending=False).head(5).round(3).to_string())
Output
Class names: {0: np.str_('malignant'), 1: np.str_('benign')} Top features pushing towards MALIGNANT (most negative): worst area -1.016 area error -0.921 worst radius -0.829 worst texture -0.797 radius error -0.789 Top features pushing towards BENIGN (most positive): fractal dimension error 0.884 mean compactness 0.836 concavity error 0.484 worst compactness 0.329 mean radius 0.259
⚠️
Read Coefficients with Care

1. Check the class order. In this dataset class 1 is benign, so positive weights point towards benign — easy to get backwards.
2. Correlated features share and swap weight. Radius, perimeter and area measure almost the same thing. That is why “worst area” pushes strongly towards malignant while “mean radius” gets a small positive weight — the model splits the effect between them. Do not read one coefficient alone when features are correlated.
3. coef_ exists only for LinearSVC and SVC(kernel='linear'). Kernel SVMs have no weight per original feature — use permutation_importance instead.

Section 11

SVM vs Other Classifiers — Comparison

Property SVM (RBF) Random Forest Logistic Reg. Neural Net
Training speed Slow: about O(n²) to O(n³) Fast (parallel) Very fast Slow
Prediction speed O(n_SV × features) Moderate Fastest Fast
Scales to large N Poorly (> 100k rows is painful) Well Very well Very well
High-dimensional sparse data Excellent (linear kernel) Poor Excellent Moderate
Feature scaling needed Essential No Recommended Yes
Probability outputs Only via extra calibration (Platt) Yes (average of tree votes) Yes, usually well calibrated Yes (softmax)
Non-linear boundaries Excellent (kernels) Excellent Linear only (unless you add features) Excellent
Hyperparameter tuning C + γ — sensitive Robust defaults Simple (C) Many — very sensitive
Interpretability Linear: moderate; RBF: low Feature importance High — coefficients Black box
Section 12

When to Use SVM

✅
Text Classification
LinearSVC with TF-IDF is one of the strongest classic text classifiers. The linear kernel does very well in the sparse, very high-dimensional space that TF-IDF creates.
spam · news topics · sentiment
✅
Small to Medium Datasets
Kernel SVMs shine on up to a few tens of thousands of rows. Training cost grows roughly with n² to n³, so it becomes a problem at large scale.
up to ~10k–50k rows (kernel SVC)
✅
Clear Margin of Separation
When the classes are separated by a clear gap, the maximum-margin boundary generalises very well, even with little data. Large-margin theory gives SVMs good generalisation guarantees.
clean binary problems
✅
More Features than Samples
SVM (with regularisation) still works when there are more features than rows — a case where many methods struggle. Gene expression data (few patients, thousands of genes) is a classic example.
genomics · bioinformatics · NLP
❌
Very Large Datasets
Kernel SVC on a million rows is impractical (hours or days). For linear problems use LinearSVC or SGDClassifier with hinge loss. For non-linear problems try gradient boosting.
use LinearSVC / SGDClassifier
❌
Heavily Overlapping Classes or Tabular Mixed Data
With strong overlap, many points become support vectors and training gets slow. On mixed tabular data, gradient boosting usually performs better with less tuning.
try XGBoost / LightGBM
Section 13

Strengths vs Weaknesses

✅ Strengths
Maximum margin — strong generalisation theory
Excellent on high-dimensional sparse data (text)
Works when features > samples
Not disturbed by points far from the boundary
The model only needs its support vectors
Kernels give flexible non-linear boundaries
Convex optimisation — one global best solution
Same idea works for regression (SVR) and outliers (One-Class SVM)
❌ Weaknesses
Kernel SVM training is slow: about O(n²) to O(n³)
Feature scaling is essential — easy to forget
No built-in probabilities — Platt scaling adds cost
Sensitive to C and γ
Hard to interpret with non-linear kernels
Painful above ~100k rows
Multi-class is built from many binary SVMs (one-vs-one in SVC)
No incremental learning in SVC (SGDClassifier can)
🔄
SGDClassifier — a Linear SVM at Scale

For data too large for SVC, use SGDClassifier(loss='hinge'). It trains a linear SVM with stochastic gradient descent, scales to millions of rows, and supports online learning with partial_fit. Use loss='modified_huber' if you also need probability estimates. 

import time
from sklearn.datasets import make_classification
from sklearn.linear_model import SGDClassifier
from sklearn.svm import LinearSVC, SVC
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import Pipeline

X, y = make_classification(n_samples=200_000, n_features=50, n_informative=20, random_state=0)
Xtr, Xte, ytr, yte = train_test_split(X, y, test_size=0.2, random_state=0)

for name, model in [
    ("LinearSVC              ", LinearSVC(C=1.0)),
    ("SGDClassifier(hinge)   ", SGDClassifier(loss="hinge", alpha=1e-4, random_state=0)),
    ("SVC(kernel='linear')*  ", SVC(kernel="linear", C=1.0)),
]:
    pipe = Pipeline([("scaler", StandardScaler()), ("m", model)])
    n = 20_000 if name.startswith("SVC") else len(Xtr)     # * SVC only gets 20k rows
    t0 = time.perf_counter()
    pipe.fit(Xtr[:n], ytr[:n])
    print(f"{name} rows={n:>7,}  fit={time.perf_counter()-t0:6.1f}s  test acc={pipe.score(Xte, yte):.4f}")
Output
LinearSVC rows=160,000 fit= 1.0s test acc=0.8039 SGDClassifier(hinge) rows=160,000 fit= 1.3s test acc=0.7991 SVC(kernel='linear')* rows= 20,000 fit= 59.6s test acc=0.8065

All three reach about 80% accuracy. LinearSVC and SGDClassifier train on 160,000 rows in about one second. Kernel-solver SVC needed a full minute for just 20,000 rows — and the time grows much faster than the number of rows.

Section 14

Real-World Applications

Bioinformatics
🧬
  • Cancer classification from gene expression
  • Protein function and structure prediction
  • Works when genes ≫ patients
  • One of the early ML successes in genomics
Image Recognition
🖼️
  • Handwritten digit recognition (MNIST)
  • Pedestrian detection with HOG features
  • Classic computer vision before deep learning
  • Still useful for small image datasets
Finance and Security
🛡️
  • Fraud detection and anomaly scoring
  • Credit risk classification
  • Intrusion detection systems
  • One-Class SVM for outlier detection
Section 15

Golden Rules

⚔️ Support Vector Machine — Golden Rules
1
Always scale features before an SVM. SVM uses distances and dot products, so a feature with range 0–10,000 completely dominates a feature with range 0–1. Put StandardScaler inside a Pipeline, so the scaler is fitted only on training data (no data leakage). In Section 08 this alone raised accuracy from 91.2% to 97.4%.
2
Tune C and γ together, on a log scale. They interact strongly: a good C at one γ can be bad at another. Use GridSearchCV with, for example, C ∈ {0.01, 0.1, 1, 10, 100} and γ ∈ {'scale', 0.001, 0.01, 0.1, 1}. Never tune one while keeping the other fixed.
3
Start with gamma='scale'. It sets γ = 1 / (n_features × X.var()), so it adapts to how spread out your (scaled) data is. 'auto' (γ = 1 / n_features) ignores the spread. After StandardScaler the two are almost the same, but 'scale' is the safer default.
4
Use LinearSVC for text and large data, SVC for small non-linear problems. LinearSVC (liblinear) scales roughly linearly with the number of rows. SVC(kernel='linear') (libsvm) grows much faster — in Section 13 it took a minute for 20,000 rows.
5
For probabilities, set probability=True when you create the model. It cannot be switched on after fitting — the model must be trained again. It runs an internal 5-fold cross-validation for Platt scaling, so training is much slower, and predict_proba can occasionally disagree with predict. Use it only when you really need probabilities.
6
If training takes too long, switch to SGDClassifier(loss='hinge') or LinearSVC. Both train a linear SVM in roughly linear time. SGD gives an approximate solution, but usually very close to the exact one. Keep StandardScaler in the pipeline as always.
7
Watch the number of support vectors. If almost every training point is a support vector, the margin is too wide or the model too simple — try a larger C or a more flexible kernel. For SVR, a very small ε also makes almost every point a support vector. Many support vectors also mean slower predictions.
8
Use One-Class SVM for anomaly detection. sklearn.svm.OneClassSVM learns the boundary of normal data without any examples of the other class. It predicts −1 for outliers and +1 for normal points. Set nu (roughly the expected fraction of outliers) instead of C. Scale the features first.