chapter 04 / cnn-vision · estimated study time 120 min
Try applying Chapter 3's MLP directly to an image: a 224×224 RGB image flattened out is 150,528 dimensions. Even if the first hidden layer needs only 1000 units, that is 150 million parameters—for one layer alone. Worse still is the structural waste:
The way out is the soul concept of this chapter: inductive bias—welding our prior knowledge about the data directly into the network's structure, so that parameters need not be wasted rediscovering common sense like "images are translationally regular and locally correlated."
A convolutional layer realizes this prior through two structural constraints:
$$ (I * K)(i, j) = \sum_{m}\sum_{n} I(i+m,\, j+n)\, K(m, n) $$A small window (the convolution kernel, e.g. the 9 weights of a 3×3 kernel) slides across the image, computing one weighted sum at each position. The meaning of the two constraints:
The key insight: the 9 numbers of a convolution kernel are the learnable weights $\theta$. Before training it is random noise; after training it "grows into" an edge detector, a texture detector, and so on. No one designs them by hand—this is precisely the watershed from traditional image processing (hand-designed Sobel operators). Below you will play the role of a "trained convolution kernel" yourself, to sense what a kernel can and cannot see.
Required experiments: ① Compare Sobel-X and Sobel-Y—watch how vertical edges and horizontal edges get "lit up" separately (only two of the square's four edges light up); ② The silence of the gradient band—in a smoothly varying region the edge-detection output is near 0; convolution detects "sudden change," not "brightness"; ③ Switch the view to ReLU—this is exactly what the next layer in a real CNN receives: negative responses are clipped away, leaving only the "detected it" signal; ④ Edit the numbers by hand—zero out the 9 weights leaving only the center at 1 (the identity kernel), then fill them in at random, and feel how an "untrained kernel" outputs meaningless noise.
The 3×3 in the middle is the kernel's 9 weights; edit the numbers directly, or use a preset. The feature map on the right uses phosphor green to represent activation strength (it is a neuron's level of excitation, not a photograph).
$N$ is the input size, $F$ the kernel size, $P$ the number of zero-padding rings (padding), $S$ the stride. Two common configurations are worth memorizing: a 3×3 kernel + $P{=}1$ + $S{=}1$ → size unchanged; $S{=}2$ → size halved (modern architectures often use it in place of pooling for downsampling).
A single kernel can find only one kind of pattern, so each layer learns dozens to hundreds of kernels in parallel. When the input has $C_{in}$ channels, each kernel is in fact a small cube of $F \times F \times C_{in}$, and the outputs stack into $C_{out}$ feature maps. The parameter count is $= F^2 C_{in} C_{out} + C_{out}$. For an RGB image, a first layer with 64 kernels: 3×3×3×64+64 = 1792.
2×2 max pooling halves the size: it keeps "whether something was detected nearby" and throws away "exactly where"—buying local translation invariance at a small cost. The more important byproduct is the expansion of the receptive field: stack 3×3 convolutions, and each neuron in the second layer indirectly sees 5×5, the third 7×7… combined with downsampling, the receptive field of deep neurons covers most of the image. Hierarchical features emerge as a result: shallow layers learn edges → middle layers assemble edges into textures and parts (eyes, wheels) → deep layers assemble parts into objects. The "layer-by-layer abstraction" thought-picture 3B1B drew in Chapter 1's video is, in a CNN, genuinely visualizable (Zeiler & Fergus 2014).
| Architecture (Year) | Layers | Epiphany |
|---|---|---|
| LeNet-5 (1998) | 5 | The basic recipe of conv + pool + fully-connected holds up; read 10% of America's checks |
| AlexNet (2012) | 8 | The same recipe × GPU × big data is enough to ignite; powered by ReLU and Dropout (this moment was covered in the prologue) |
| VGG (2014) | 19 | The architecture can be minimal: only 3×3 from start to finish, depth is justice |
| GoogLeNet (2014) | 22 | Parallel multi-scale branches (Inception), with 1×1 convolutions for channel compression to save compute |
| ResNet (2015) | 152 | Residual connections break the depth wall, an error rate of 3.6% first surpassing humans (5.1%)—see the next section |
After VGG, everyone naturally kept stacking deeper, and then hit a wall: a plain 56-layer network had higher training error than a 20-layer one. Note this is not overfitting (the training error itself got worse), it is optimization failure—in theory 56 layers could at least learn the extra 36 layers as identity mappings to match the 20-layer network, but gradient descent cannot find that solution. The gains of depth were eaten up by the difficulty of optimization.
Kaiming He's solution was to "build a road" for optimization. A plain block requires the layers to learn the target mapping $H(x)$ directly; a residual block instead learns the correction $F(x) = H(x) - x$, with output:
$$ y = F(x) + x $$Two immediate effects:
x = x + attn(norm(x)) yourself), and GPT is essentially a stack of residual blocks. "An LLM is the spiritual heir of ResNet" is no exaggeration. The other half of the recipe is BatchNorm (already derived in Chapter 3): after the convolution, before the activation—the conv→BN→ReLU trio.The Vision Transformer of 2020 posed a provocative question: is the inductive bias of convolution a necessity or a crutch? Its approach was brutal yet elegant: cut the image into 16×16 patches, flatten each patch and linearly project it into a vector—treating a patch as a "visual word"—then feed them to a standard Transformer (the star of Chapter 6) and let self-attention learn the relationships among patches on its own.
The result perfectly echoes the bias-variance framework of Chapter 1: on small and medium datasets ViT loses to CNNs (no translation prior, high variance); once the data reaches the hundred-million scale ViT overtakes them—when there is enough data, the model can learn a "prior" better than any human-designed one, and the hard-coded bias instead becomes a ceiling. This is yet another confirmation of "The Bitter Lesson" (Sutton's two-page manifesto in the appendix's sources): general methods + more compute and data win over human priors in the long run. The landscape in 2026: for pure vision tasks CNNs (including ConvNeXt) and ViTs coexist, but the vision encoders of multimodal LLMs are almost uniformly of the ViT family—because they share the same architectural language as the language tower.
import torch, torch.nn as nn
class ResidualBlock(nn.Module):
"""conv→BN→ReLU→conv→BN, then add the shortcut, and finally ReLU together"""
def __init__(self, ch_in, ch_out, stride=1):
super().__init__()
self.conv1 = nn.Conv2d(ch_in, ch_out, 3, stride, padding=1, bias=False)
self.bn1 = nn.BatchNorm2d(ch_out) # BN has its own bias, so conv can drop bias
self.conv2 = nn.Conv2d(ch_out, ch_out, 3, 1, padding=1, bias=False)
self.bn2 = nn.BatchNorm2d(ch_out)
# when size/channels change, the shortcut uses a 1×1 conv to align the shape, otherwise identity
self.shortcut = nn.Identity()
if stride != 1 or ch_in != ch_out:
self.shortcut = nn.Sequential(
nn.Conv2d(ch_in, ch_out, 1, stride, bias=False),
nn.BatchNorm2d(ch_out))
def forward(self, x):
out = torch.relu(self.bn1(self.conv1(x)))
out = self.bn2(self.conv2(out))
return torch.relu(out + self.shortcut(x)) # ← y = F(x) + x, the soul of the whole chapter is in this line
# verify the output-size formula: O = (N + 2P - F)/S + 1
x = torch.randn(1, 64, 56, 56)
print(ResidualBlock(64, 64)(x).shape) # → (1, 64, 56, 56) size unchanged
print(ResidualBlock(64, 128, stride=2)(x).shape) # → (1, 128, 28, 28) halved and channels increased