chapter 10 / rag · estimated study time 120 min
Chapter 6 said it: a model's factual knowledge lives mainly in the FFN weights. This kind of parametric knowledge has three structural defects:
RAG (Retrieval-Augmented Generation) solves this by turning a closed-book exam into an open-book one: before answering, it first retrieves relevant material from an external knowledge base and stuffs it into the context, so the model "cites" rather than "recalls." The pipeline in one breath: document → chunk → embed → store; query → embed → nearest-neighbor retrieval → (rerank) → assemble prompt → generate. Every link has technical detail; we unpack them one by one.
The core abstraction was already laid down in Chapter 5: semantic similarity = closeness in vector space. An embedding model maps arbitrary text to a dense vector (768–3072 dimensions), trained with contrastive learning: pull related pairs together (question↔answer, title↔body) and push unrelated pairs apart (other samples in the same batch serve as negatives). The loss is InfoNCE (cross-entropy's twin sibling from Chapter 2—a "find the positive within the batch" classification):
$$L = -\log \frac{e^{\text{sim}(q, d^+)/\tau}}{\sum_{d \in \text{batch}} e^{\text{sim}(q, d)/\tau}}$$Similarity uses cosine (equivalent to the dot product after normalization). Two engineering points: ① the two-tower (dual-encoder) structure—query and document are each encoded independently, so document vectors can be precomputed offline, and at query time you only compute the query embedding once plus a nearest-neighbor lookup (this is why RAG can be fast); ② its cost is that query and document have no interaction, so its accuracy ceiling is lower than a cross-encoder that concatenates the two and runs them through the model together—a trade-off we cash in during the reranking of §6.
18 toy documents are laid out on a 2D "semantic map" (a cartoon version of embedding space), with 4 preset queries. Must-do experiments: ① Query "can I get my money back"—keyword mode returns zero results (the corpus contains no exact words "money" or "back"), while semantic mode lands precisely on the refund cluster: this is the "lexical gap" that embeddings solve; ② Query "what perks do VIPs get"—"VIP" never appears in the corpus, yet semantic mode still drops right into the center of the membership cluster; ③ Query "how do I export my pixel artwork," then switch between the two modes—both hit: keyword retrieval is not dead, which is why production systems use hybrid retrieval.
Cyan Q = query · green numbered dots = Top-k hits · gray dots = unmatched documents · note the recipe/weather distractor clusters in the center
Brute-force scanning a billion 1024-dimensional vectors to find the nearest neighbor: ~4TB of memory reads per query, infeasible. Production uses approximate nearest neighbors (ANN), and the mainstream choice is HNSW (Hierarchical Navigable Small World graph):
How you cut up a document determines the semantic integrity of the retrieval unit—get chunking wrong, and everything downstream is wasted:
The engineered version of Lab experiment ③: semantic retrieval understands synonymous rewrites but is actually worse than BM25 keyword retrieval for exact strings (model numbers, error codes, person names). The production standard is two-path recall + fusion + reranking:
RAG is a pipeline, so evaluation must attribute by segment (the RAG version of Chapter 8's evaluation-first methodology):
| Dimension | Question | Metric |
|---|---|---|
| Retrieval quality | Did it find what it should have found? | Recall@k / MRR (requires annotated query→relevant-chunk pairs) |
| Faithfulness | Is the answer based only on the retrieved content? | LLM-as-judge, checking citations sentence by sentence (the RAGAS approach) |
| Answer relevance | Did it answer the actual question? | LLM-as-judge / manual spot checks |
A failure-mode checklist (diagnose by pipeline position, with the most common remedy attached):
"The context is already 1M tokens—can't I just stuff all the documents in?" Three reasons keep RAG alive: cost (Chapter 9 just did the math: you pay the prefill bill for the whole library's tokens on every request, and the KV cache eats VRAM; retrieval only pays for the top-5 chunks); accuracy (lost in the middle is worse at a million tokens—needle-in-a-haystack benchmarks look good, but multi-needle reasoning still drops the ball); permissions and freshness (the retrieval layer naturally supports per-user ACL filtering and second-level incremental updates, which stuffing the context cannot do). In practice it's a spectrum: for documents under 50K tokens, stuff them directly (it's even cheap with caching, Chapter 9); for large libraries, use RAG; and the two are often mixed.
The direction of evolution is Agentic RAG: retrieval is no longer a fixed "one query, one generation" pipeline, but rather making search a tool for the Agent—the model decides for itself whether to search, what to search, and how many rounds, rewriting the query and searching again if it's unhappy with the results, and reasoning across documents in multiple hops. This is precisely the topic of the next chapter: once an LLM is given tools and a loop, every pipeline becomes its own decision.
import numpy as np
from sentence_transformers import SentenceTransformer, CrossEncoder
# ---- Offline: build the index (the document side of the two-tower, precomputed once) ----
docs = open('knowledge.md').read().split('\n## ') # structure-aware chunking: split by level-2 headings
embedder = SentenceTransformer('BAAI/bge-m3') # Chinese-English bilingual embedding
doc_vecs = embedder.encode(docs, normalize_embeddings=True) # (n, 1024); after normalization, dot product = cosine
# ---- Online: retrieve + rerank + generate ----
def retrieve(query, k_recall=20, k_final=5):
q = embedder.encode([query], normalize_embeddings=True)
sims = doc_vecs @ q.T # one matrix multiply = cosine similarity over the whole library
cand = np.argsort(-sims[:, 0])[:k_recall] # recall top-20 (brute-force scan at toy scale; swap in HNSW to scale up)
reranker = CrossEncoder('BAAI/bge-reranker-v2-m3') # cross-encoder fine-grained ranking
scores = reranker.predict([(query, docs[i]) for i in cand])
return [docs[cand[i]] for i in np.argsort(-scores)[:k_final]]
def answer(query):
chunks = retrieve(query)
context = '\n\n'.join(f'[Source {i+1}] {c}' for i, c in enumerate(chunks))
prompt = (f"Answer based only on the materials below. For information not in the materials, "
f"state clearly 'not mentioned in the materials', and cite the source numbers.\n\n{context}\n\nQuestion: {query}")
return llm(prompt) # any LLM API; note: putting context first lets you hit the prompt cache (Chapter 9)
print(answer("How long until a refund arrives?")) # → Per [Source 2], 3-5 business days, refunded by the original route.