RAG Build & Break

Retrieval quality caps answer quality — so this lesson makes you watch it cap.

Your win for this lesson: a small retrieval-augmented pipeline you built, then deliberately broke by mis-chunking — with a table showing grounding collapse as chunking degrades. The rubric's second M2 exercise. Prerequisite: Lesson 0002 (RAG + failure modes).

1 · The pipeline, minimally

RAG is: chunk a corpus → retrieve the relevant chunks for a question → inject them into context → answer from them (Lesson 0002). Build the smallest honest version — no vector DB required for the lesson; keyword/overlap scoring makes the failure modes just as visible and keeps the mechanism in view.

Corpus: one longish document with distinct facts in different sections — a fake product FAQ or policy doc, ~2–3k words, where answers depend on specific passages.
Chunk: split into passages; start with a sane strategy (by paragraph, or ~200 words with ~40-word overlap).
Retrieve: score chunks against the question (keyword overlap is fine), take the top k (start k=3).
Answer: inject the top-k chunks into the prompt with the instruction "Answer only from the passages below; if they don't contain the answer, say so." Cite which chunk each claim came from.

Pass: ask five questions whose answers live in different sections; a well-chunked pipeline answers them with correct citations, and correctly says "not in the passages" for a question you know isn't covered.

2 · Break the chunking — the lab

Now degrade chunking on purpose and re-run the same five questions. Each degradation is a real production mistake:

Chunking	Correct (n/5)	Failure you saw
Sane (baseline)
Too small (~20 words, no overlap)
Too large (whole doc = 1 chunk)
Zero overlap on sentence boundaries
Split mid-fact (fixed 100-char, ignore words)

Predicted failures to watch for: too small severs a fact from its context ("the limit is 50" — 50 what?); too large reintroduces context rot (Lab A, Lesson 0006) and buries the answer; zero overlap drops facts that straddle a boundary; mid-fact splits retrieve half a number.

Pass: at least two chunking strategies measurably lowered the correct-answer count, and you can name the mechanism for each. The model didn't get dumber between runs — retrieval did.

The M2 intuition, proven: bad chunking and weak retrieval poison answers more often than the model does, and no amount of reasoning recovers a fact retrieval failed to supply. When a grounded agent gives wrong answers, debug the pipeline before you touch the prompt or the model — a habit that becomes an M3 reflex (traces before prompts) and an M4 SLO (retrieval quality as a first-class metric).

3 · One knob that isn't chunking

Hold chunking sane and vary k (how many chunks you inject): try k=1, 3, 8, 20. Note the tradeoff — too few misses the answer (recall), too many reintroduces distraction and cost (Lab A again). There's a sweet spot, and it's task-specific.

Pass: you found a k that beats both k=1 and k=20 on your five questions, and can explain why each extreme lost.

Self-grade

Pipeline works — chunk → retrieve → grounded answer with citations, and an honest "not covered"
Grounding collapse shown — ≥2 chunking degradations lowered accuracy, each with a named mechanism
k tradeoff found — a middle k beats both extremes, explained
Debug reflex stated — you said retrieval, not the model, is the first suspect for a grounded agent's wrong answers

Lesson 0010 — episodic memory: the agent recalls a past run and changes its plan; then the M2 readiness check.

RAG Build & Break

1 · The pipeline, minimally

2 · Break the chunking — the lab

3 · One knob that isn't chunking

Self-grade

Next