chapter 09 / llm-inference · estimated study time 120-150 min
The mine planted in Chapter 6 now officially explodes: training can run in parallel over an entire sentence, but generation must proceed token by token, serially. A single inference actually contains two phases with completely opposite personalities:
| Prefill (reading the prompt) | Decode (generating token by token) | |
|---|---|---|
| Compute pattern | The entire prompt processed in parallel in one matrix multiply | Each step computes only 1 token |
| Bottleneck | Compute-bound (saturates GPU compute) | Memory-bandwidth-bound: every token generated requires hauling all the weights from VRAM into the compute units once |
| User perception | Time to first token (TTFT) | Generation speed (tokens/s) |
Decode's bandwidth ceiling can be computed in your head directly: tokens/s ≈ VRAM bandwidth ÷ model bytes. An H100 (3.35TB/s) running a bf16 70B (140GB): single-stream ceiling ≈ 24 tokens/s—no card, however expensive, can break physics. This roofline intuition explains half the techniques in this chapter: quantization (making the model bytes smaller), batching (hauling the weights once to serve multiple requests), speculative decoding (hauling the weights once to verify multiple tokens)—all of them wrestle with the single act of "hauling the weights."
Naive generation recomputes attention over the entire history for every new token—turning $O(n^2)$ complexity into $O(n^3)$ over the whole generation, which is unacceptable. Observation: under a causal mask, the K and V of historical tokens never change (they depend only on the content before themselves). So store them: each time a token is generated, compute only its own q/k/v, append the new k/v to the cache, and have attention query the cache directly. The KV footprint per token has a must-memorize formula:
$$\text{KV bytes/token} = \underbrace{2}_{K\text{ and }V} \times L \times n_{kv} \times d_{head} \times \underbrace{2}_{bf16}$$LLaMA-3 8B ($L{=}32, n_{kv}{=}8, d_{head}{=}128$): 128KB/token—a single sequence at 8K context is 1GB, and at 128K context it is 16GB (larger than the model weights). The real cost of long context lies not in compute but in VRAM, and it grows linearly with batch size—this is why "concurrency" became a core parameter of inference serving.
Must-do experiments: ① 8B + 128K context—the KV is larger than the weights; ② switch to GPT-3-style MHA—the horrifying bill of 96 unshared KV heads makes it instantly obvious why GQA became standard; ③ switch to DeepSeek MLA—at the same context the KV shrinks by an order of magnitude, which is the hardware reason it can serve long context cheaply.
Formula: 2 × L × n_kv × d_head × 2 bytes × context × concurrency · verdict includes a rough estimate with weights
Inference serving before 2023 had a hidden waste: the KV cache was pre-allocated as contiguous VRAM at the "maximum possible length," while only a small portion was actually used—VRAM utilization was often below 40%. vLLM's PagedAttention brought over the whole approach of OS virtual memory: the KV is split into fixed-size blocks (e.g., 16 tokens/block), logically contiguous but physically scattered, allocated on demand. Combined with continuous batching (instead of waiting for an entire batch to finish, whenever a sequence ends a new request is swapped in immediately, so the GPU never sits idle), throughput improves 2-4×. Today vLLM/SGLang are the de facto standard for open-source inference serving—between "running a model" and "efficiently serving a model" lies an entire discipline of systems engineering.
The roofline already said it: decode speed ≈ bandwidth ÷ model bytes. Halve the bytes and speed roughly doubles while VRAM halves—quantization is inference's most cost-effective lever:
| Scheme | Bit width | Key points |
|---|---|---|
| bf16 | 16 | Baseline |
| FP8 | 8 | Natively supported on H100, works for both training and inference, nearly lossless—the new default at frontier labs |
| INT8 (LLM.int8) | 8 | Per-channel scaling + outlier separation (LLM activations have a few enormous outlier dimensions; those whole columns are kept in fp16) |
| GPTQ / AWQ | 4 | Post-training weight quantization: GPTQ minimizes second-order error layer by layer; AWQ protects important weights according to activation magnitude. ~1% loss for 4× compression |
| KV cache quantization | 8/4 | The second battlefield beyond weights—huge gains in long-context scenarios (directly cuts the bill from §2) |
Rule of thumb: 4-bit weights are the last stop before the quality cliff (below 3 bits the quality drops noticeably); more parameters + aggressive quantization usually beats fewer parameters + full precision (70B-int4 ≳ 13B-bf16).
Decode is bandwidth-bound → the GPU compute units sit largely idle → could we "haul the weights once and verify multiple tokens"? Speculative decoding: a small draft model quickly guesses k tokens, and the large model verifies them in parallel in one forward pass. The key is that it is mathematically lossless: each guessed token is accepted with probability $\min(1, p_{target}/p_{draft})$, and at a rejection point it resamples from the corrected distribution—it can be proven that the final output distribution is exactly identical to that of the large model sampling on its own. If the draft is right you gain k× speed; if it is wrong you only waste a little idle compute. Typical speedup is 2-3×, especially in domains like code where "the next token is often quite certain." Variants: self-drafting (Medusa/EAGLE add lightweight prediction heads to the large model), n-gram matching (copying the answer from the prompt).
Once the model gives the next-token distribution, how do you choose? Greedy (always take the maximum) gets trapped in repetition loops ("I think I think I think"—high-probability phrases reinforcing each other). The practical combo:
In a scaled product, model selection is essentially unit economics. Three levers, in order of priority:
Must-do experiments: ① default parameters (100,000 DAU, remix 1/100, PGC 1/1000, cache hit 70%) and compare the monthly bills of the three model tiers; ② pull the cache hit to 0 and then to 95%—understand why "input structure design" is valuable; ③ pull the remix rate from 1/100 to 1/10 (the product blows up and everyone is creating)—watch the flagship model's bill spiral out of control, and why self-hosted small models are an order-of-magnitude difference in high-frequency scenarios.
Monthly cost = 30 × daily generations × (missed input × base price + hit input × cache price + output × output price) · all sliders draggable