chapter 08 / llm-alignment · estimated study time 180 min
What you get when pretraining ends is not ChatGPT—it is an internet document simulator: ask it "What is the capital of China?" and it might continue with "What is the capital of Japan? What is the capital of Korea?"—because in the corpus this sentence most often appears in a list of exam questions. It holds an enormous amount of knowledge, but its personality is "the next line of a random web page." Post-training has to solve three things: format (learning conversational turns), helpfulness (answering rather than continuing), and harmlessness (refusing dangerous requests). The roadmap is the InstructGPT trilogy: SFT → reward model → RL, plus the post-2023 substitutes and upgrades (DPO, GRPO, RLVR).
Mechanically SFT offers nothing new—it is still next-token-prediction cross-entropy; only the data changes, from "random web pages" to "carefully written conversational demonstrations," and the loss is computed only over the answer portion (the prompt portion is masked out). The conversation is first serialized through a chat template:
<|im_start|>system
You are a pixel-art generator that only outputs valid JSON.<|im_end|>
<|im_start|>user
Draw an 8x8 orange cat<|im_end|>
<|im_start|>assistant
{"palette": ["#000", "#f80", ...], "pixels": [[0,1,...], ...]}<|im_end|> ← only this segment counts toward lossA real case study that runs through this chapter and Chapter 13: getting a 4B-9B small model to reliably output pixel-art JSON (palette + pixel matrix). Why is this a clever task design?
The standard methodology of the loop (no step can be skipped):
Full fine-tuning of an 8B model needs ~112GB of training memory (weights + gradients + Adam state). LoRA's insight: the weight change $\Delta W$ caused by fine-tuning is low-rank—task adaptation only requires a small rotation on top of the original capability. So freeze $W$ and train only a low-rank-decomposed side branch:
$$W' = W + \frac{\alpha}{r}\, B A, \qquad B \in \mathbb{R}^{d \times r},\ A \in \mathbb{R}^{r \times d},\ r \ll d$$$A$ is Gaussian-initialized and $B$ is zero-initialized (guaranteeing $\Delta W = 0$ at $t=0$, starting from the base model with no perturbation). With $d{=}4096, r{=}16$, the trainable parameters of one $d\times d$ matrix drop from 16.8M to $r(d+d) = 131$K—0.8%. Gradients and Adam state are stored only for that 0.8%, so memory plummets from 14 bytes/parameter to 2 bytes/parameter (frozen weights) plus a tiny remainder. Once training is done you can merge $BA$ back into $W$ for zero inference overhead; or keep it as a standalone adapter (tens of MB), hot-swapping multiple tasks onto one base model.
Required experiments: ① LLaMA-3 8B default config—see the cliff between full 112GB and LoRA 16GB; ② select 70B and see which hardware can run QLoRA; ③ pull r from 16 to 128—trainable parameters grow 8x, but memory barely moves (the bulk is frozen weights), to understand "LoRA's memory bottleneck is not r"; ④ check the MLP modules—parameters multiply several-fold, the standard operation for knowledge-injection tasks.
Memory is a rough weight-side estimate (add 10-20% for activations/batch) · Mac unified memory is scaled by available ratio · ✓ = it fits
SFT's ceiling: a demonstrator can only show what is "good," not tell the model "how much better A is than B," yet many goals (helpful, honest, tactful) cannot be written as demonstrations and can only be compared. RLHF turns comparison into a loss:
① Reward model (RM): collect human preferences over answer pairs $(y_w \succ y_l)$, and use the Bradley-Terry model to tie "preference probability" to "score difference," then maximize likelihood:
$$P(y_w \succ y_l) = \sigma\big(r_\phi(x, y_w) - r_\phi(x, y_l)\big), \qquad L_{RM} = -\log \sigma\big(r_\phi(x,y_w) - r_\phi(x,y_l)\big)$$② PPO optimizes the policy: have the model generate answers, the RM scores them, but a KL penalty must be added to anchor to the reference model:
$$\max_\pi\ \mathbb{E}_{y \sim \pi}\Big[ r_\phi(x, y) \Big] - \beta\, \mathbb{D}_{KL}\big[\pi(y|x)\, \|\, \pi_{ref}(y|x)\big]$$RLHF-PPO is engineering-heavy (serving 4 models at once: policy/reference/RM/value function). DPO (2023) found a mathematical shortcut whose derivation is just three steps—worth walking through in full:
Step one: the KL-constrained optimization problem above has an analytic solution (a standard result of the calculus of variations):
$$\pi^*(y|x) = \frac{1}{Z(x)}\, \pi_{ref}(y|x)\, \exp\!\big(r(x,y)/\beta\big)$$Step two: solve back for the reward—$r(x,y) = \beta \log \frac{\pi^*(y|x)}{\pi_{ref}(y|x)} + \beta \log Z(x)$. The reward can be expressed by the policy itself!
Step three: substitute into the Bradley-Terry loss, and the partition function $Z(x)$ cancels exactly when the two answers are subtracted:
$$L_{DPO} = -\log \sigma\Big( \beta \log \tfrac{\pi_\theta(y_w|x)}{\pi_{ref}(y_w|x)} - \beta \log \tfrac{\pi_\theta(y_l|x)}{\pi_{ref}(y_l|x)} \Big)$$No need to train an RM, no sampling, no PPO—gradient descent directly on the preference data, as simple as SFT, yet implicitly doing the same RL optimization. The cost: it is an offline method that does not learn on its own generated trajectories (the ghost of exposure bias from Chapter 5), and its ceiling is usually lower than online RL. A common combination in practice: DPO for a cheap first round of alignment, online RL (PPO/GRPO) for the fine polish.
GRPO (DeepSeek) cut the heaviest component from PPO—the value network (critic). Advantage estimation switches to in-group relative comparison: sample $G$ answers for the same prompt, and each one's advantage is its score standardized within the group:
$$A_i = \frac{r_i - \text{mean}(r_1..r_G)}{\text{std}(r_1..r_G)}$$"Better than the group average is reinforced, worse is suppressed"—no need to learn a value function to serve as a baseline, saving half the memory and all the critic-tuning black magic.
RLVR (reinforcement learning with verifiable rewards) solves the problem at the other end: the RM is a lossy proxy, but math problems can be checked against the answer, and code can run tests—the reward is programmatic and unhackable. The recipe (the R1 route): a set of verifiable tasks + GRPO + enough compute, and the model spontaneously emerges long chains of thought, reflection, and self-correction ("wait, let me recheck this"), with no human-demonstrated reasoning process needed. This is the engine of o1/R1 reasoning models, and it opens up the second curve of test-time scaling (paying off the prologue's foreshadowing: the longer it thinks, the more accurate it gets).
# 0) Eval first: run the base model baseline (assume eval.py validates JSON/schema/palette compliance)
python eval.py --model Qwen/Qwen3-8B --suite pixel_art_v1
# → baseline: 0/12 pass ← remember this number, it's the reference frame for every conclusion
# 1) Data: jsonl, one chat-format sample per line (~a few thousand, quality > quantity)
# {"messages":[{"role":"system","content":"You are a pixel-art generator…"},
# {"role":"user","content":"Draw an 8x8 orange cat"},
# {"role":"assistant","content":"{\"palette\":[…],\"pixels\":[…]}"}]}
# 2) LoRA training (mlx-lm, 600 steps ≈ 3 epochs, a pipeline-verification run)
python -m mlx_lm lora \
--model Qwen/Qwen3-8B \
--train --data ./data \
--num-layers 16 --batch-size 2 --iters 600 \
--learning-rate 1e-5 # SFT learning rate: two orders of magnitude smaller than pretraining
# 3) Evaluate against the baseline immediately after training (controlled-experiment discipline)
python eval.py --model ./adapters --suite pixel_art_v1
# → 9/12 pass (0→9: the pipeline works; look at the remaining 3 failures one by one, decide to add data or change the schema)
# 4) Merge the adapter and export for deployment (or keep the adapter for hot-swapping)
python -m mlx_lm fuse --model Qwen/Qwen3-8B --adapter-path ./adaptersA quick scale reference: a ~13M-token dataset, an 8B model, high-end Apple Silicon ≈ 3-4 hours per epoch—a complete SFT is an overnight job. This rhythm of "build the dataset and eval by day, run training at night, check the score in the morning" is the daily life of post-training for a vertical team.
Video walkthrough: the second half of Karpathy's overview covers exactly all the topics of this chapter (you already watched the first half in Chapter 7):