chapter 05 / sequence-models · estimated study time 120 min · the last stop before the Transformer
The models in the first four chapters carry a hidden assumption: the input is fixed-length, and its dimensions have no ordering (images have spatial structure, but a fixed size). Language, speech, and time series break both of these:
The CNN's answer was weight sharing across space; the sequence model's answer is exactly analogous: weight sharing across time—the same set of parameters is reused at every time step. This is the recurrent neural network.
An RNN maintains a hidden state $\mathbf{h}_t$ (the network's "working memory"), updating it each time it reads a word:
$$\mathbf{h}_t = \tanh\big(W_h \mathbf{h}_{t-1} + W_x \mathbf{x}_t + \mathbf{b}\big)$$The same $W_h, W_x$ are used at all time steps—processing 3 words and 300 words uses the same handful of parameters (variable length solved), and "a rule learned at position 7" automatically applies at position 70 (time-shift sharing). Unroll the recurrence over time and the RNN is a special deep network whose depth equals the sequence length and whose layers share weights—this viewpoint is the key to understanding the disaster in the next section.
Training an RNN still uses backpropagation, only propagated back along the unrolled time axis, hence the name BPTT (Backpropagation Through Time). The problem hides in "long-distance credit assignment": the gradient of the loss at step $t$ with respect to the hidden state $k$ steps earlier, by the chain rule:
$$\frac{\partial \mathbf{h}_t}{\partial \mathbf{h}_k} = \prod_{i=k+1}^{t} \frac{\partial \mathbf{h}_i}{\partial \mathbf{h}_{i-1}} = \prod_{i=k+1}^{t} W_h^\top\, \text{diag}\big(\tanh'(\mathbf{z}_i)\big)$$The same matrix $W_h$ multiplied together $t-k$ times. Let its largest singular value be $\sigma_{\max}$: when $\sigma_{\max} < 1$ the gradient vanishes exponentially with distance—"Xiaoming" 50 steps away has an effect on the current gradient of roughly zero, and long-range dependencies cannot be learned; when $\sigma_{\max} > 1$ it explodes exponentially—the loss becomes NaN. This shares its origin with the depth-direction vanishing gradient of Chapter 3, but is worse: in a deep network the layers are different matrices, so luck can cancel out; an RNN is the same matrix raised to a power, its fate entirely determined by the spectrum of $W_h$, with no reprieve.
The LSTM (1997) surgical plan: in addition to $\mathbf{h}_t$, add a cell state $\mathbf{c}_t$—a dedicated long-term memory conveyor belt—and use three learnable "gates" to control reading and writing. A gate is just a sigmoid-output vector of values in 0~1, multiplied element-wise to implement a "soft switch":
$$\mathbf{f}_t = \sigma(W_f [\mathbf{h}_{t-1}, \mathbf{x}_t] + \mathbf{b}_f) \qquad \text{forget gate: how much old memory to keep}$$ $$\mathbf{i}_t = \sigma(W_i [\mathbf{h}_{t-1}, \mathbf{x}_t] + \mathbf{b}_i), \qquad \tilde{\mathbf{c}}_t = \tanh(W_c [\mathbf{h}_{t-1}, \mathbf{x}_t] + \mathbf{b}_c) \qquad \text{input gate: how much new info to write}$$ $$\boxed{\ \mathbf{c}_t = \mathbf{f}_t \odot \mathbf{c}_{t-1} + \mathbf{i}_t \odot \tilde{\mathbf{c}}_t\ } \qquad \text{cell state update}$$ $$\mathbf{o}_t = \sigma(W_o [\mathbf{h}_{t-1}, \mathbf{x}_t] + \mathbf{b}_o), \qquad \mathbf{h}_t = \mathbf{o}_t \odot \tanh(\mathbf{c}_t) \qquad \text{output gate: how much to expose to this step's output}$$Fix your eyes on the boxed update: the backbone of $\mathbf{c}_t$ is addition. Taking the gradient, $\frac{\partial \mathbf{c}_t}{\partial \mathbf{c}_{t-1}} = \text{diag}(\mathbf{f}_t)$—no more repeated multiplication by $W_h$! As long as the forget gate learns to stay open ($f \approx 1$), the gradient can flow along the cell state through hundreds of steps without decay.
The 2014 Seq2Seq chains two LSTMs in a relay, solving tasks where "both input and output are variable-length sequences" (translation, summarization): the encoder finishes reading the source sentence and compresses all its understanding into the final hidden state $\mathbf{h}_{enc}$; the decoder takes that as its initial state and generates the target sentence word by word. Elegant, but with a congenital disability:
Bahdanau attention (2015) grants the decoder exactly this right. The encoder keeps the hidden state at every position $\mathbf{h}_1,\dots,\mathbf{h}_n$ (no longer keeping only the last one); before the decoder generates the $t$-th word, it takes three steps:
① Score—take the current decoding state $\mathbf{s}_{t-1}$ and ask each source position, "are you relevant to what I'm about to generate now?":
$$e_{tj} = \mathbf{v}^\top \tanh\big(W_s \mathbf{s}_{t-1} + W_h \mathbf{h}_j\big) \qquad j = 1,\dots,n$$② Normalize—softmax turns the scores into weights (summing to 1):
$$\alpha_{tj} = \frac{\exp(e_{tj})}{\sum_{k} \exp(e_{tk})}$$③ Weighted aggregation—mix the information from all source positions by weight to obtain a context vector dedicated to this step:
$$\mathbf{c}_t = \sum_{j=1}^{n} \alpha_{tj}\, \mathbf{h}_j$$The bottleneck disappears: every time it generates a word, the decoder custom-builds on the spot a summary of the source sentence. When generating "yesterday," $\alpha$ concentrates on "昨天"; when generating "park," it concentrates on "公园"—the alignment is learned by the model itself from the translation data, with no human dictionary whatsoever.
Below is a simulated, well-trained attention model (Chinese→English). Required experiments: ① Hover over "yesterday"—its attention reaches across the whole sentence to align with the second word "昨天" (Chinese and English word order differ, and this is exactly where attention beats "align by position"); ② Hover over "the"—a function word has no clear counterpart, so attention is diffuse—the model's way of expressing "uncertainty"; ③ Drag the temperature to 0.1—soft alignment degenerates into a hard pointer; drag it to 5—it degenerates into a uniform average (≈ a Seq2Seq with no attention). The temperature is the $T$ from the softmax in Chapter 2, and you'll meet it a third time in LLM sampling in Chapter 9.
Hover/click an English word · the depth of the highlight on the Chinese word = attention weight α · the bar chart shows the exact value
Training Seq2Seq involves a subtle choice: what do you feed as the decoder's input at step $t$? Using the model's own output from the previous step—one wrong word and everything afterward is trained on garbage, and it cannot be parallelized; so in practice we use teacher forcing: during training always feed the true previous word. The price is exposure bias: the model never sees its own mistakes during training, so at inference time, once it goes wrong it enters a completely unfamiliar distribution of states and the errors snowball.
This 2015 chestnut is still alive in the LLM era: GPT pretraining is large-scale teacher forcing (every position is conditioned on the true preceding text), and the snowball effect of hallucination is related to it; and part of the reason the RLHF/RLVR of Chapter 8 works is precisely that reinforcement learning trains the model on its own generated trajectories—it can be seen as a systematic fix for exposure bias. One thread, pulled for a decade.
import torch, torch.nn as nn
class LSTMCell(nn.Module):
"""Merge the four gate computations into one big matmul (the engineering standard); each line maps to §4's formulas"""
def __init__(self, d_in, d_h):
super().__init__()
self.W = nn.Linear(d_in + d_h, 4 * d_h) # f, i, c̃, o computed in one shot
self.d_h = d_h
def forward(self, x, h, c):
z = self.W(torch.cat([x, h], dim=-1))
f, i, c_tilde, o = z.chunk(4, dim=-1)
f = torch.sigmoid(f) # forget gate
i = torch.sigmoid(i) # input gate
c_tilde = torch.tanh(c_tilde) # candidate memory
o = torch.sigmoid(o) # output gate
c = f * c + i * c_tilde # ★ additive backbone: a residual connection in the time dimension
h = o * torch.tanh(c)
return h, c
# Run a variable-length sequence, verify the state shapes
cell = LSTMCell(d_in=32, d_h=64)
h = torch.zeros(1, 64); c = torch.zeros(1, 64)
for t in range(100): # after 100 steps the gradient can still flow back to t=0 (thanks to the additive backbone)
h, c = cell(torch.randn(1, 32), h, c)
print(h.shape, c.shape) # torch.Size([1, 64]) ×2
# In production just use nn.LSTM (an internal fused cuDNN implementation, an order of magnitude faster):
# rnn = nn.LSTM(input_size=32, hidden_size=64, num_layers=2, batch_first=True)