Discrete Convolution
Press Next → or use ← → arrow keys
The Story Behind Convolution
The flashlight is the kernel — a little grid of weights. Reusing the same detector at every position is exactly why convolution is so powerful and so parameter-efficient.
What Discrete Convolution Does
A kernel slides over the input. At every position it computes a single dot product between its weights and the local patch — one output number per position.
Instead of a separate weight for every pixel, one small kernel is shared across the entire input. A 3×3 kernel has just 9 weights — no matter how big the image.
Convolution vs Cross-Correlation
Here's the honest truth every deep-learning course glosses over: what CNNs call "convolution" is really cross-correlation.
PyTorch's nn.Conv2d, TensorFlow's Conv2D, and essentially every framework implement cross-correlation but call it convolution. The network learns the right weights either way, so nobody flips.
1-D Worked Example
Input [1, 2, 3, 4, 5], kernel [1, 0, −1] (a simple edge detector), stride 1, no padding. The kernel slides three times:
Output size = 5 − 3 + 1 = 3. Every window gives −2 because the input rises at a constant slope — the detector reports the same steady gradient everywhere.
The Kernel as a Feature Detector
Different weight patterns detect different things. A few classic 3×3 kernels:
These hand-designed kernels are just for intuition. In a real network the weights are learned by backpropagation — the model discovers whichever detectors best solve the task.
2-D Convolution Over an Image
For images the kernel is a small grid that slides in two directions. Each placement produces one cell of the feature map:
Multiply the window by the kernel element-wise, sum it up, and write the result into the top-left of the feature map. Slide one step and repeat until the whole image is covered.
2-D Worked Example — Finding a Vertical Edge
A 4×4 image with a bright stripe down the middle, convolved with a vertical-edge kernel [[1,0,−1],[1,0,−1],[1,0,−1]]:
A negative response flags a dark-to-light transition, positive a light-to-dark one. The kernel has literally located both sides of the stripe.
Stride & Padding
Two knobs control how the kernel moves and what happens at the borders:
Without padding, pixels at the very edge are visited by the kernel far less often than central ones, and the image shrinks with every layer. Padding preserves size and gives borders a fair say.
The Output-Size Formula
One formula predicts the feature-map size from the input, kernel, padding, and stride:
| Case | Setting | Output |
|---|---|---|
| "Same" padding, stride 1 | P = (F−1)/2 | O = N (unchanged) |
| No padding, stride 1 | P = 0 | O = N − F + 1 |
| 28×28 input, 3×3 kernel | P = 0, S = 1 | 26 × 26 |
A mismatched output size is the most common CNN bug. Plug the numbers into the formula before wiring layers together — it takes five seconds and saves an hour.
Convolution From Scratch
The entire 2-D operation is a double loop of dot products:
import numpy as np
def conv2d_scratch(x, k, stride=1, padding=0):
"""x: (H,W) input, k: (Fh,Fw) kernel"""
if padding > 0:
x = np.pad(x, padding, mode='constant')
H, W = x.shape
Fh, Fw = k.shape
Oh = (H - Fh) // stride + 1
Ow = (W - Fw) // stride + 1
out = np.zeros((Oh, Ow))
for i in range(Oh):
for j in range(Ow):
patch = x[i*stride : i*stride+Fh,
j*stride : j*stride+Fw]
out[i, j] = np.sum(patch * k) # dot product
return outReal frameworks vectorise it heavily for speed, but the logic is identical: slide, multiply, sum. Everything else in a CNN is stacking this operation.
Putting It Together — The CNN Layer
A convolutional layer isn't one kernel — it's K kernels applied at once, each producing its own feature map. Stack those maps and you get the layer's output volume.
Early layers detect edges and textures; deeper layers combine them into parts and whole objects. More channels carry more distinct detectors as the spatial resolution drops.
Everything in One Table
| Concept | What it is |
|---|---|
| Kernel / filter | Small grid of learned weights — the detector |
| Feature map | Grid of dot-product outputs — where the pattern appears |
| Stride | Step size of the slide; larger = smaller output |
| Padding | Zero border; "same" preserves size, "valid" shrinks |
| Output size | O = ⌊(N − F + 2P)/S⌋ + 1 |
| Weight sharing | One kernel reused everywhere — few parameters |
| What libraries do | Cross-correlation (no flip), called "convolution" |
Master slide-multiply-sum and everything else is a parameter choice around it. That's the whole of discrete convolution.
5 Golden Rules
Slide a small detector everywhere, share its weights, and learn what to look for. From edge detection to ImageNet, it all starts with discrete convolution.