chapter 03 / deep-learning · estimated study time 150-180 min

Foundations of Deep Learning
Functions That Learn Features

Paying Off a Setup: XOR and the Ceiling of Linearity

In the Chapter 2 lab, no matter how long we trained logistic regression on the XOR data, it stalled at 50% accuracy—its hypothesis space contains only straight lines. At the time we offered three ways out: hand-craft features, kernel methods, or let the model learn the features itself. The first two share a problem: the features (or kernel) are chosen by a human, and one wrong choice dooms everything. The third path is the neural network:

$$\text{Logistic regression:}\ \hat{p} = \sigma(\mathbf{w}^\top \mathbf{x} + b) \qquad\Longrightarrow\qquad \text{Neural network:}\ \hat{p} = \sigma\big(\mathbf{w}_2^\top\, \underbrace{g(W_1 \mathbf{x} + \mathbf{b}_1)}_{\text{learned new features } \mathbf{a}}\, + b_2\big)$$

Structurally it is just "first apply a linear transformation plus a nonlinearity $g$, then do logistic regression." But its significance is revolutionary: the hidden layer $\mathbf{a} = g(W_1\mathbf{x}+\mathbf{b}_1)$ is the feature the model learns on its own. For XOR, the network discovers for itself two intermediate features like "are $x$ and $y$ on the same side"—and in this new space, XOR becomes linearly separable. The entire secret of deep learning condenses into one sentence: learn better and better representations layer by layer, until a final linear model wraps things up.

The Multilayer Perceptron and the Universal Approximation Theorem

Generalize the structure above: an $L$-layer network, where layer $\ell$ is $\mathbf{a}^{(\ell)} = g\big(W^{(\ell)} \mathbf{a}^{(\ell-1)} + \mathbf{b}^{(\ell)}\big)$. The theoretical guarantee comes from the Universal Approximation Theorem (Cybenko 1989 / Hornik 1991): a single-hidden-layer network, given enough hidden units, can approximate any continuous function on a compact set to arbitrary precision.

Backpropagation: The Full Derivation

Training is still the same old trio: model (above), loss (cross-entropy), optimization (gradient descent). The only new problem: how do we compute the gradients of millions of parameters quickly? The answer is the chain rule + dynamic programming, and it is called backpropagation.

Deriving a Two-Layer Network Step by Step

Let the forward pass be: $\mathbf{z}_1 = W_1\mathbf{x}+\mathbf{b}_1,\ \mathbf{a}_1 = g(\mathbf{z}_1),\ z_2 = \mathbf{w}_2^\top\mathbf{a}_1 + b_2,\ \hat p = \sigma(z_2)$, with the loss $L$ being cross-entropy. Working backward from the output, we apply the chain rule layer by layer:

Step one (a gift from Chapter 2, carried straight over):

$$\delta_2 \equiv \frac{\partial L}{\partial z_2} = \hat p - y$$

Step two: the gradient of the output-layer parameters. The partial derivative of $z_2$ with respect to $\mathbf{w}_2$ is just $\mathbf{a}_1$:

$$\frac{\partial L}{\partial \mathbf{w}_2} = \delta_2\, \mathbf{a}_1, \qquad \frac{\partial L}{\partial b_2} = \delta_2$$

Step three (the key one): the error travels back through the weights and the activation function to the hidden layer. $z_2$ depends on $\mathbf{a}_1$, and $\mathbf{a}_1$ depends on $\mathbf{z}_1$:

$$\delta_1 \equiv \frac{\partial L}{\partial \mathbf{z}_1} = \underbrace{\delta_2\, \mathbf{w}_2}_{\text{error distributed by weights}} \odot \underbrace{g'(\mathbf{z}_1)}_{\text{times the activation's derivative}}$$

Step four: the gradient of the hidden-layer parameters, in exactly the same form as step two:

$$\frac{\partial L}{\partial W_1} = \delta_1\, \mathbf{x}^\top, \qquad \frac{\partial L}{\partial \mathbf{b}_1} = \delta_1$$

See the pattern? Each layer's gradient = that layer's error $\delta$ × that layer's input—the full-grown form of "error × input" from Chapters 1 and 2. And the propagation of the error $\delta$ has a single unified recurrence (valid for a network of any depth):

$$\boxed{\ \delta^{(\ell)} = \big(W^{(\ell+1)\top} \delta^{(\ell+1)}\big) \odot g'(\mathbf{z}^{(\ell)})\ }$$

This is the entire content of the word "backpropagation": run the forward pass once and store each layer's activations ($O(n)$), then run the recurrence backward to carry $\delta$ from the output layer back to the input layer (another $O(n)$). Computing the gradients of all parameters costs only about twice a single forward pass, independent of the number of parameters—without this property, training a GPT with trillions of parameters would be flat-out impossible. PyTorch's loss.backward() is, internally, running exactly this recurrence over the computation graph.

Interactive Lab: Train a Neural Network with Your Own Hands

Below is a 2-H-1 network that genuinely runs backpropagation right in your browser (the code is exactly the four-step derivation from §3; you can read the assets/neural.js source to compare). Must-do experiments: ① Train XOR with H=1—no matter how many parameters, it cannot learn it (one hidden unit can fold space only once); ② Bump H up to 4 and retrain—watch how the decision boundary "bends" out two diagonal regions; ③ Spiral data + H=8 + tanh, witnessing the limits of a small network; ④ Switch the activation to sigmoid and retrain on the spiral, feeling the crawling slowness caused by vanishing gradients; ⑤ Crank the learning rate to the max and watch the loss explode. The depth of the background color = the model's confidence; the pale zones are its "hesitation regions."

H=2 is theoretically enough to solve XOR, yet in experiments training sometimes fails and gets stuck around 75%. Why? Try "Reinitialize" several times before answering.

This is a live demonstration of §2's point that "a good solution existing ≠ gradient descent finding it." At H=2 the loss surface has bad local minima / flat regions, and an unlucky random initialization falls into them; at H=4 the parameter space is more redundant, with far more "downhill paths," and it succeeds almost every time. This is an important and counterintuitive phenomenon in deep learning: overparameterization (having more parameters than theoretically needed) actually makes optimization easier—large models are not only more expressive, they are also easier to train.

A History of Activation Functions

Function	Formula	Drawback / Advantage
sigmoid	$1/(1+e^{-z})$	Saturates at both ends, $\sigma' \le 0.25$; in deep networks the repeated multiplication makes gradients vanish exponentially; output is not zero-centered
tanh	$\tanh(z)$	Zero-centering improves on sigmoid, but it still saturates
ReLU	$\max(0, z)$	Derivative is constantly 1 over the positive region, a gradient highway; computation is nearly free. The cost: the "dying ReLU" (once a neuron falls into the negative region its gradient is 0 and it never wakes up again)
GELU	$z\cdot\Phi(z)$	A smooth version of ReLU, the choice of GPT/BERT
SwiGLU	$\text{Swish}(W_1 x)\otimes W_2 x$	A gated structure, standard issue for the FFN of LLaMA and most LLMs since (see Chapter 6)

Why is ReLU the unsung hero behind deep learning's takeoff? Look at the $\delta$ recurrence: every step back multiplies by $g'$ once. With sigmoid's $g' \le 0.25$, after passing back 10 layers the gradient has shrunk by at least $4^{10} \approx$ a million times—the bottom layers learn essentially nothing (vanishing gradients). On the active path, ReLU has $g' = 1$, so the error passes through losslessly. The AlexNet paper devoted a whole figure to showing that ReLU converges 6× faster than tanh—among the three ingredients of the 2012 revolution, the "algorithm" item was mainly this.

The Evolution of Optimizers: From SGD to AdamW

Plain SGD has two pain points: ① when the loss surface is a long, narrow canyon, the gradient oscillates back and forth in the steep direction and crawls in the gentle one; ② all parameters share one learning rate, yet the gradient magnitudes of different parameters may differ by orders of magnitude. Two lines of improvement ultimately merge into Adam:

Clue One: Momentum

$$\mathbf{v}_t = \beta\, \mathbf{v}_{t-1} + (1-\beta)\, \mathbf{g}_t, \qquad \theta_{t+1} = \theta_t - \eta\, \mathbf{v}_t$$

Take an exponential moving average of past gradients: oscillating directions cancel out positive against negative, while a consistent direction rolls faster and faster—like a ball with mass rolling downhill. $\beta=0.9$ is roughly averaging the last 10 steps.

Clue Two: Adaptive Learning Rates (RMSProp)

$$\mathbf{s}_t = \beta_2\, \mathbf{s}_{t-1} + (1-\beta_2)\, \mathbf{g}_t^2, \qquad \theta_{t+1} = \theta_t - \frac{\eta}{\sqrt{\mathbf{s}_t} + \epsilon}\, \mathbf{g}_t$$

Divide each parameter by the moving root-mean-square of its own gradient magnitude: a parameter whose gradient is always large automatically slows down, one whose gradient is small automatically speeds up—each parameter gets a personally tailored learning rate.

Confluence: Adam (2014) = Momentum + RMSProp + Bias Correction

$$\hat{\mathbf{m}}_t = \frac{\mathbf{m}_t}{1-\beta_1^t}, \quad \hat{\mathbf{s}}_t = \frac{\mathbf{s}_t}{1-\beta_2^t}, \qquad \theta_{t+1} = \theta_t - \eta\, \frac{\hat{\mathbf{m}}_t}{\sqrt{\hat{\mathbf{s}}_t} + \epsilon}$$

Bias correction solves the cold-start problem: $\mathbf{m}_0 = 0$ makes the early estimate biased toward zero, and dividing by $(1-\beta^t)$ corrects it (once $t$ is large that term tends to 1 and automatically deactivates).

Initialization: The Starting Point Decides the Fate

• All-zero initialization = a symmetry catastrophe: all neurons in a layer have identical forward outputs, identical backward gradients, and stay identical forever—equivalent to having only one neuron. Symmetry must be broken randomly.
• Too large: activations enter the saturation region (sigmoid/tanh) or explode; too small: the signal decays to zero layer by layer.
• Xavier initialization (for tanh): $\text{Var}(w) = \frac{2}{n_{in}+n_{out}}$, keeping the variance of both forward activations and backward gradients conserved across layers. He initialization (for ReLU): $\text{Var}(w) = \frac{2}{n_{in}}$—ReLU cuts away half the activations, so the variance must be doubled to compensate. The derivation idea for both is "make $\text{Var}(z_{out}) = \text{Var}(z_{in})$."

Regularization and Normalization

Regularization: Three Tricks Against Overfitting

• L2 / weight decay: add $\frac{\lambda}{2}\|\theta\|^2$ to the loss, favoring small weights = a smoother function. Bayesian view: equivalent to a Gaussian prior on the parameters.
• Dropout: during training, randomly zero out neurons with probability $p$, forcing features to be redundant and not to rely on specific collusion paths. Two readings: ① each mini-batch trains a different sub-network, so inference amounts to an ensemble of exponentially many sub-networks (a cheap version of the random-forest idea); ② it injects multiplicative noise into the hidden layers. At inference nothing is dropped and outputs are scaled by the expectation. It is still used in the Transformer era (attention/residual dropout).
• Early stopping: stop as soon as the validation loss starts rising again. Implicit regularization—it limits the distance the parameters travel from the initial point.

Normalization: The Engineering Pillar That Makes Deep Networks Trainable

$$\text{Norm}(x) = \gamma\,\frac{x - \mu}{\sqrt{\sigma^2 + \epsilon}} + \beta$$

The same formula; the only difference is over what you compute $\mu, \sigma$: BatchNorm goes along the batch dimension (the same channel across samples)—standard issue for vision, but it depends on batch statistics: it is unstable with small batches, requires the running mean from training at inference, and is awkward when sequence length varies. LayerNorm goes along the feature dimension (within a single sample)—independent of the batch, naturally friendly to sequences, which is exactly why the Transformer chose it. RMSNorm further drops the mean subtraction, dividing only by the root-mean-square; after LLaMA it became the LLM default (it will appear in the GPT you implement from scratch in Chapter 6). Their common effect: pull each layer's input back to a stable distribution, making the loss surface smoother and allowing a large learning rate.

Hands-On Code: Backprop by Hand in NumPy vs. PyTorch

On the left is a line-by-line translation of the §3 derivation; on the right is industrial reality—the two must produce identical gradients:

python · backprop_from_scratch.py

import numpy as np
rng = np.random.default_rng(0)

# XOR data
X = np.array([[0,0],[0,1],[1,0],[1,1]], dtype=float)
y = np.array([0.,1.,1.,0.])

H = 4
W1 = rng.normal(0, np.sqrt(2/2), (H, 2)); b1 = np.zeros(H)   # He initialization
W2 = rng.normal(0, np.sqrt(2/H), H);      b2 = 0.0
sig = lambda z: 1/(1+np.exp(-z))

for epoch in range(5000):
    # ---- forward (store intermediates, the backward pass needs them) ----
    Z1 = X @ W1.T + b1            # (4,H)
    A1 = np.maximum(0, Z1)        # ReLU
    z2 = A1 @ W2 + b2             # (4,)
    p  = sig(z2)
    # ---- backward: the four steps of §3, line by line ----
    d2  = (p - y) / len(X)        # δ₂ = p̂ - y          (step one)
    gW2 = A1.T @ d2; gb2 = d2.sum()         #            (step two)
    d1  = np.outer(d2, W2) * (Z1 > 0)       # δ₁ = δ₂W₂ ⊙ g'(step three)
    gW1 = d1.T @ X;  gb1 = d1.sum(0)        #            (step four)
    # ---- SGD update ----
    lr = 0.5
    W1 -= lr*gW1; b1 -= lr*gb1; W2 -= lr*gW2; b2 -= lr*gb2

print(np.round(p, 3))   # → about [0, 1, 1, 0], XOR solved ✓

python · the same thing in PyTorch

import torch, torch.nn as nn

X = torch.tensor([[0.,0.],[0.,1.],[1.,0.],[1.,1.]])
y = torch.tensor([[0.],[1.],[1.],[0.]])

model = nn.Sequential(nn.Linear(2, 4), nn.ReLU(), nn.Linear(4, 1))
opt = torch.optim.AdamW(model.parameters(), lr=0.05, weight_decay=0.01)
loss_fn = nn.BCEWithLogitsLoss()   # fused sigmoid+cross-entropy operator (numerically stable, the §2 setup)

for epoch in range(2000):
    opt.zero_grad()
    loss = loss_fn(model(X), y)
    loss.backward()                # ← automatically runs the §3 recurrence, a.k.a. autograd
    opt.step()

print(torch.sigmoid(model(X)).detach().round().squeeze())  # tensor([0.,1.,1.,0.])

The closing video is this chapter's "final feast": Karpathy starts from an empty Python file and hand-writes an entire automatic-differentiation engine, micrograd—after watching it, loss.backward() will hold no mystery for you whatsoever. It is the first stepping stone toward "implementing GPT from scratch" in Chapter 6:

Chapter Quiz