The Critic/Verifier Step

Agents fail silently and confidently. A critic converts silent failures into caught ones — and you measure by how many.

Your win for this lesson: a verifier stage added to your agent and a measured reduction in silent wrong answers — the rubric's third Level 3 exercise — then the Level 3 readiness check. Prerequisites: Lesson 0003, Lesson 0014 (you need the eval set).

1 · Establish the baseline

Run your agent across a test set that includes hard cases where it sometimes produces plausible-but-wrong answers (the dangerous Level 3 failure — Lesson 0003). Using your 0014 evals, record the silent wrong rate: answers that were wrong but looked confident and well-formed (no crash, no "I'm not sure").

Pass: you have a baseline number — e.g. "4 of 20 answers were confidently wrong." If your agent never errs on your set, make the set harder; you can't measure a reduction you can't first produce.

2 · Add the verifier

Insert a self-correction stage before the agent returns (Lesson 0003, Cluster C):

Reflection / critic: a second model call — ideally fresh-context, given only the question and the proposed answer — prompted to find flaws: "Check this answer. Does it follow from the evidence? Recompute any arithmetic. If it's wrong or unsupported, say so and why." A fresh-context verifier catches what a self-critique in the same window rationalizes.
Loop or gate: if the critic flags a problem, the agent revises (bounded retries) or returns an honest "uncertain" rather than a confident wrong answer.

3 · Measure the delta

Configuration	Correct	Silent wrong	Honest "uncertain"	Extra tokens/run
Baseline (no critic)				—
+ critic/verifier

Pass: silent wrong answers dropped, and you can state the cost — the critic adds tokens and latency, and may convert some right answers to "uncertain" (a real tradeoff). A verifier that halves silent-wrong at 1.5× cost is a decision you can now articulate, not just an instinct.

The Level 3 arc, complete: single-agent-default and the multi-agent bar (0013), tracing and trajectory evals (0014), self-correction measured (0015). You've turned "agents fail silently" from a warning into an instrumented, mitigated, quantified property of your system.

4 · The Level 3 readiness check

Bring to your teaching agent: a failing agent you can debug from traces alone (they'll hand you a broken run — or you replay a 0014 plant — and you find the fault by reading), and your eval suite catching a planted regression. Both are the literal Level 3 gate.

Trace-first debugging: given a broken run, you reach for the trace, not the prompt, and locate the faulting step by reading
Eval as gate: your suite goes red on a regression, with a trajectory assertion doing the catching on at least one case
Architecture judgment: you can argue for/against adding an agent on this task with trace evidence, not vibes
Silent-failure literacy: you can name the difference between a crash and a confidently-wrong answer, and which your instruments catch

Pass and Level 3 closes — the Level 4 hands-on arc (SLO dashboard, guardrails, HITL, full improvement loop) is the ownership stage.

Lesson 0016 — the SLO dashboard: put your live agent under production targets.

The Critic/Verifier Step

1 · Establish the baseline

2 · Add the verifier

3 · Measure the delta

4 · The Level 3 readiness check

Next