chapter 07 / llm-pretraining · estimated time 150-180 min
Formally, pretraining is disappointingly simple—the softmax cross-entropy from Chapter 2, with about 100,000 token classes, maximizing the log-probability of the next token at every position in the corpus:
$$L = -\frac{1}{T}\sum_{t=1}^{T} \log P_\theta\big(x_t \mid x_{<t}\big)$$The depth lies in "what kind of text appears in the corpus." Predicting "Paris is the ____ of France" requires facts; predicting "therefore x = ____" requires having done the preceding algebra; predicting "the murderer is ____" on the last page of a detective novel requires digesting the whole book's clues. Next-token prediction is a superset of tasks: push the loss low enough and grammar, facts, and reasoning are all squeezed out together—not because the model "wants" to learn them, but because without them the loss won't go down.
The model doesn't read characters, it reads tokens. The BPE (Byte Pair Encoding) algorithm is as naive as a homework problem: start from 256 bytes, repeatedly find the most frequent adjacent pair in the corpus and merge it into a new token, until the vocabulary reaches the target size (GPT-4 is about 100,000):
Corpus: "low lower lowest" vocab starts from bytes
Round 1: most frequent pair = (l,o) → merge "lo" vocab +1
Round 2: most frequent pair = (lo,w) → merge "low" vocab +1
Round 3: (low,e) → "lowe" …… until |V| = targetVocabulary size is a three-way trade-off: a large vocab → shorter sequences (saving $O(n^2)$ attention) but a larger embedding matrix and undertrained rare tokens; a small vocab is the reverse. Many of an LLM's "stupid moments" are actually the tokenizer taking the blame: miscounting how many r's are in "strawberry" (it sees the three atoms [st][raw][berry], not 10 letters), unstable arithmetic ("1234" might be split into [12][34], making digit-aligned addition impossible), and Chinese costing 1-2 tokens per character versus ~1.3 tokens per word for English (the same context window holds less Chinese information and the API bills more). Karpathy's blunt claim is worth remembering: "Every quirk of an LLM, traced to its root, is half the tokenizer's fault."
In 2020, Kaplan et al. found that loss follows a power law in parameter count $N$, data size $D$, and compute $C$, holding across seven orders of magnitude:
$$L(N) \propto N^{-0.076}, \qquad L(D) \propto D^{-0.095}, \qquad C \approx 6ND$$(Where $C \approx 6ND$ comes from: the forward pass is about 2 FLOPs per parameter per token (one multiply, one add), the backward pass is about 2× the forward, totaling 6.) Kaplan's conclusion of "prioritize scaling N" led to the GPT-3 era's "big model, little data" (175B fed only 300B tokens). In 2022, Chinchilla corrected the experimental methodology (retuning the learning rate schedule at each compute point), and the conclusion flipped: for a given compute budget, N and D should be scaled in proportion, with an optimal ratio of about $D^* \approx 20N$. The Chinchilla model with 70B parameters × 1.4T tokens comprehensively beat the 4× larger Gopher—rendering half the field's training plans obsolete on the spot.
Required experiments: ① Click the five presets in order and watch the D/N verdict travel from "undertrained" (GPT-3, 1.7:1) to "inference-optimal" (LLaMA-3, 1875:1)—the historical trajectory; ② Pull N up to 1T and set D to 20N—see how many H100s and how many years it takes to train a trillion-parameter model Chinchilla-optimally, and understand why trillion-scale models are almost all MoE; ③ Verify by mental arithmetic at fixed compute: double N, halve D, C stays the same—but the loss differs, which is exactly the question scaling laws are meant to answer.
Sliders are log-scale · MFU assumed at 40% (real large clusters are 35-45%) · MoE models compute C from active parameters
Using FineWeb (Hugging Face's openly reproduced pretraining data pipeline; the appendix sources have report links) as a template, going from Common Crawl's ~hundreds of PB of raw web pages to 15T clean tokens, every step is an order-of-magnitude cut:
The key discipline: train essentially only ~1 epoch (when there's enough data, the returns from seeing the same data again drop sharply and memorization worsens). Contrast Chapter 1's classical ML with hundreds of epochs—the main battleground for LLM overfitting is not the epoch count but data repetition and contamination. The exhaustion of high-quality data (the "data wall") is a real problem for 2025-2026, and the responses are: synthetic data (generated by a strong model + filtered), multimodal data, and the RL of Chapter 8 (trading compute for data efficiency).
70B parameters × (weights 2 bytes + gradients 2 + Adam's two moments 8) ≈ 840GB—a single 80GB GPU can't hold even a fraction. Four knives, distinguished by "what gets sliced":
| Strategy | What gets sliced | Cost |
|---|---|---|
| Data Parallel DP | Slice the batch; each GPU holds a full model replica | All-reduce gradients each step; no memory saved |
| ZeRO / FSDP | Shard optimizer states/gradients/weights across GPUs, temporarily gathering when needed | Trades communication for memory; the memory cure for DP |
| Tensor Parallel TP | Slice a single matrix multiply (by row/column); several GPUs jointly compute one layer | Two all-reduces per layer; needs NVLink-class bandwidth, generally kept within a node |
| Pipeline Parallel PP | Slice into segments by layer; different GPUs handle different layers | Pipeline "bubbles" (head and tail idle time), filled with micro-batches |
Ten-thousand-GPU training is 3D parallelism: TP within a node (eating NVLink bandwidth) × PP across nodes × overall DP/ZeRO. Plus two standard fixtures: mixed precision (compute in bf16, accumulate in fp32—bf16 has the same exponent bits as fp32, so it doesn't overflow easily, and it has largely retired fp16+loss scaling) and gradient checkpointing (don't store intermediate activations, recompute them in the backward pass—trading 1/3 extra compute for more than half the activation memory, enabled by default in almost all large-model training).
MoE (Mixture of Experts) replicates each layer's FFN into $E$ "experts," with a router picking the top-$k$ (e.g., 1-2 out of 8) to activate for each token. The effect: parameter count ×E, while the compute per token is almost unchanged. DeepSeek-V3: 671B total parameters, only 37B activated per token—buying the knowledge capacity of 671B with the compute of 37B. But there's no free lunch; the trade-offs need careful accounting:
| Dense | MoE | |
|---|---|---|
| Inference compute | All parameters participate | Only the top-k experts are activated; low FLOPs |
| Memory | = parameter count | All experts must be loaded into memory (routing is per-token, and you can't predict which one the next token will use)—compute is saved, memory is not |
| Training engineering | Mature, simple | A notch harder: load-balancing auxiliary loss (to prevent expert collapse—a few experts monopolizing traffic), all-to-all communication for expert parallelism, capacity-factor tuning |
| Fine-tuning | Works out of the box | Routing easily becomes imbalanced on small data; framework support is uneven |
The landmine planted in Chapter 6: attention is $O(n^2)$ in sequence length—at 128K context, the attention matrix alone is an astronomical figure. An active line of research tries to compress it to $O(n)$: linear attention / state space models (the Mamba, RWKV, DeltaNet families), whose idea can be crudely summarized as "compress the KV history into a fixed-size recurrent state"—in a sense a return to the RNN (the ghost of Chapter 5) while keeping the Transformer's training parallelism. The retrieval ability ("needle in a haystack") of pure linear models still lags full attention, so the mainstream compromise of 2025-2026 is hybrid architectures: most layers use linear attention, with a full-attention layer inserted every few layers as a backstop (Qwen3.5, MiniMax, etc. all fall into this category).
A typical LLM pretraining configuration, every item of which you derived in earlier chapters:
optimizer: AdamW(β₁=0.9, β₂=0.95, weight_decay=0.1) # Ch.3; β₂ lowered to handle gradient noise
lr: warmup 2000 steps → cosine decay to 10% of peak # Ch.1 cure for the SGD noise floor
grad_clip: 1.0 # Ch.5 gradient clipping
precision: bf16 (fp32 master weights) # this chapter §6
batch: ~4M tokens/step (millions) # Ch.1 mini-batch in its extreme formThe closing video: Karpathy's 3.5-hour overview ties this chapter (pretraining) and the next (post-training) into a complete picture; after watching it you'll have an engineering-level understanding of "how ChatGPT is made":