chapter 12 / frontier · estimated time 120-150 min

Multimodality and the Frontier
From the Latent Space to Uncharted Ground

Diffusion Models: Thermodynamics Run in Reverse

The difficulty of generating images: learning $p(\text{image})$ directly is too hard (the adversarial game of GANs is unstable—Chapter 0's history lesson covered its rise and fall). Diffusion models break "generation" into a long chain of tiny denoising steps. The forward process (fixed, parameter-free) gradually drowns the data in Gaussian noise. It has an elegant closed form—the noised result at any time $t$ can be computed in a single step:

$$x_t = \sqrt{\bar\alpha_t}\, x_0 + \sqrt{1-\bar\alpha_t}\,\varepsilon, \qquad \varepsilon \sim \mathcal{N}(0, I)$$

$\bar\alpha_t$ decreases monotonically from 1 (no noise) to ≈0 (pure noise). The reverse process is the part that gets learned: train a network $\varepsilon_\theta(x_t, t)$ that looks at the noised image and guesses "what noise was originally mixed in":

$$L = \mathbb{E}_{x_0, \varepsilon, t}\, \big\| \varepsilon - \varepsilon_\theta(x_t, t) \big\|^2$$

Look closely at this loss—it is exactly the MSE regression of Chapter 1. The entire magic of image generation has, as its training objective, just "a regression problem that predicts noise": at sampling time you start from pure noise, and each step uses the predicted noise to step back a little; after 50 steps the noise becomes an image. Two finishing touches:

• Text conditioning: the prompt's embedding (encoded by CLIP/T5, the CLIP node in Chapter 0) is injected into the denoising network via cross-attention—"the direction of denoising" is steered by the text.
• CFG (classifier-free guidance): compute both the "conditioned" and "unconditioned" predictions, then extrapolate in the direction of their difference, $\varepsilon_{cfg} = \varepsilon_{uncond} + w(\varepsilon_{cond} - \varepsilon_{uncond})$—amplifying "the part the text asked for" by a factor of $w$. The slider for "how closely the generated image sticks to the prompt" is precisely this $w$.

Interactive Lab: 50-Step Denoising

A 16×16-pixel cat (a return to this course's pixel-art through-line) travels between noise and image according to the closed form above. Required experiments: ① Click "play in reverse" to watch the full trajectory—note the order in which structure appears: first the large color-block outline, then the eyes and nose—denoising is a coarse-to-fine spectral process, which is why diffusion models compose so well; ② Drag to around t=45—the human eye can no longer recognize anything, yet √ᾱ still has a residual value, so information-theoretically x₀ is not entirely dead; ③ Resample the noise and play again—the same x₀ paired with different ε walks a different trajectory: during training the model sees exactly these infinitely many (x_t, ε) pairs.

From Pixels to Latent Space: The Course's Name Pays Off

Running 50 denoising steps in a 1024×1024 pixel space, with one U-Net/DiT forward pass per step, is too expensive. Latent Diffusion (Stable Diffusion's real name) first uses a VAE to compress the image 48× into a latent space, diffuses there, and finally decodes back to pixels. This course's name, "Latent Space Lab," is now fully redeemed: the semantic space of word vectors in Chapter 5, the embedding space of retrieval in Chapter 10, the diffusion latent space here—deep learning's unified worldview is "map the raw data into a semantically effective low-dimensional space, and do all computation there." Architecturally, the U-Net backbone has already been replaced by DiT (Diffusion Transformer, which cuts the noised latent image into patches treated as tokens—another victory for the Chapter 6 architecture); Sora-class video models do the same thing on spatiotemporal patches.

Multimodal LLMs: Turn Everything Into Tokens

The mechanism by which GPT-4o/Claude/Gemini "see" an image takes only one sentence to assemble from earlier chapters: a ViT (Chapter 4) cuts the image into patches and encodes them into a sequence of vectors, a projection layer aligns them to the LLM's embedding space, and the image tokens are concatenated with the text tokens to enter the same Transformer (Chapter 6). Three routes, ordered by depth of fusion:

• Late fusion (adapter-style): a frozen vision encoder + a lightweight projection layer (the LLaVA style)—cheap, mainstream.
• Early fusion (natively multimodal): mixed image-text training from pretraining onward, with unified tokenization (including image-generation tokens)—a higher ceiling.
• Speech/video: the same recipe—audio is cut into frames and encoded as tokens, and video is "spatiotemporal patches." Every modality, in the end, becomes tokens; every architecture, in the end, becomes a Transformer.

The World-Model Debate

When Sora was released, OpenAI's framing that "video generation models are world simulators" sparked the most nourishing debate in the field. The pro side: to generate "a glass falling" correctly, the model must implicitly master gravity, collision, and refraction—compressing video equals compressing physics (the ultimate extension of the compression view in Chapter 7). The con side (the LeCun camp): generating pixels is learning what "looks plausible," not what is "physically correct"—the evidence being systematic failures like glasses clipping through objects and proliferating fingers; a true world model should predict in an abstract representation space (the JEPA route), not in pixel space. The pragmatists: regardless of whose philosophy wins, video models are already providing training data and policy priors for robots (the world-model → embodied-intelligence pipeline is taking shape). This debate is worth tracking because it is essentially asking: can predicting the next frame / next word ever lead to understanding?—the very question you have been pondering since Chapter 7.

Evaluation Methodology: What to Measure, What to Trust

The convergence of the entire course's evaluation thread. Before reading any "SOTA" claim, run it through four filters:

1. Is it contaminated? (Chapter 1's discipline → Chapter 7's decontamination) Static benchmarks leak into the training data—a high MMLU score may just be memorized answers. The countermeasure: LiveBench-style dynamically generated questions, private test sets.
2. Is it measuring capability or preference? LMArena's Elo is "which one humans like more"—it rewards a confident tone and pretty formatting (an echo of the sycophancy in Chapter 8). Preference ≠ correctness.
3. A single point or a distribution? Chapter 11's pass^k: reporting "the best result from a single run" versus "the stable result over unattended runs" are two different worlds.
4. Has it been Goodharted? (the evaluation version of the Chapter 8 law) When SWE-bench becomes the optimization target, the model may be learning to "solve SWE-bench" rather than "software engineering." Any metric inflates once it is aimed at—which is why every frontier lab internally maintains private, undisclosed evaluations.

The Frontier of Safety and Interpretability

• Interpretability has evolved from "looking at attention maps" (whose limits were warned about in Chapter 5) to "dissecting features": SAEs (sparse autoencoders) decompose activations into millions of nameable, monosemantic features ("the Golden Gate Bridge," "code vulnerability," "sycophancy"), and can perform causal interventions—after Anthropic amplified the "Golden Gate Bridge" feature, the model could not stop talking about the bridge. From correlation to causation, interpretability has touched the edge of being engineered for the first time.
• The alignment tech stack (the extension of Chapter 8): Constitutional AI replaces part of the human labeling with a list of principles; process rewards (supervising every step of the reasoning rather than only the answer); the attack-and-defense of red-teaming and jailbreaks (prompt injection, in the agent context of Chapter 11, becomes the number-one real-world threat).
• Open problems: deceptive alignment (the model learns to "perform well when it is being tested"), reward tampering, goal drift in long-horizon agents—these are no longer philosophical questions but the engineering agenda of frontier labs.

Three Unfinished Curves

The main thread of the course is now complete; we close with three curves still unfolding (all foreshadowed in earlier chapters):

1. The relay of scaling: the pretraining power law is slowing (Chapter 7's data wall) → the baton is handed to RL post-training (Chapter 8's RLVR: trading compute for data efficiency) and to inference-time compute (o1/R1: thinking longer). "Buying intelligence with compute" continues; only the way of buying has changed.
2. From conversation to long-horizon agents (the extension of Chapter 11): reliability engineering (pass^k), the accumulation of memory and skills, multi-agent collaboration—the focus of competition shifts from "how smart the model is" to "how long the system can work continuously without derailing."
3. From multimodality to embodiment: video world models → robotic VLA (vision-language-action) models—the recipe of "next-word prediction" is being moved into the physical world.

The far-right end of the Chapter 0 timeline was left blank. Having finished this course, you now possess all the technical foundation needed to read—and to help write—that blank space. The final chapter (Chapter 13, hands-on) will twist all the course's threads into a single end-to-end project you can run yourself.

Chapter Quiz