Tracing & the Planted Regression

You debug agents by reading traces. So first you build the trace — then an eval that catches a bug you hid.

Your win for this lesson: full tracing on your agent, plus an eval set that catches a regression you deliberately introduced — the rubric's second M3 exercise, and the core of the M3 gate. Prerequisite: Lesson 0003.

1 · Instrument a real trace

A trace is the full trajectory of a run: every model call, tool call, and retrieval, with inputs, outputs, and timing (Lesson 0003). Extend the logging you've built since 0005 into one structured trace per run — one record per step:

{"run_id", "step", "type": "model_call|tool_call",
 "input", "output", "tokens", "latency_ms", "ts_offset"}

Hosted tools (Langfuse, Arize, Opik) do this for you in production and add a UI — Vernant already has a langfuse-bridge in the tree if you want to wire the real thing. But build the JSONL version by hand first: the discipline is knowing what must be captured, and you only learn that by capturing it.

Pass: after one run you can reconstruct exactly what the agent did — and, critically, why it made each decision — from the trace alone, without re-running. If you can't tell why it chose a tool, your trace is missing the model's input for that step.

2 · Build a small eval set

Write 8–12 test cases for your agent: input + expected outcome. Then split scoring into the two M3 levels — because a right answer via a broken path breaks at scale (Lesson 0003):

Outcome checks: is the final answer correct? (assert on the value)
Trajectory checks: did it take a sane path? (e.g. "used the calculator tool for arithmetic", "made ≤ N model calls", "never read a note it didn't search for first") — assertions over the trace from step 1.

Pass: your suite runs and passes green on the current agent, and at least a third of your assertions are trajectory-level, not just final-answer.

3 · Plant a regression — the lab

Introduce a bug that a naive outcome-only test would miss. Best kind: one that keeps answers right on easy cases but corrupts the path or fails on edge cases. Examples:

Weaken the system prompt so the agent sometimes does mental math instead of calling the tool (outcome often still correct — trajectory now wrong).
Cap the loop at 2 iterations (multi-hop questions now truncate).
Silently drop is_error handling (errors now corrupt instead of recover).

Pass: your eval suite goes red on the planted regression — and the assertion that catches it is a trajectory check for at least one of your plants, proving why trajectory evals earn their keep. If an outcome-only suite would have stayed green, you've demonstrated the exact M3 intuition.

What you now own: the two instruments of M3. Traces turn "it broke" into "here's the step it broke at." Evals turn "seems fine" into "the suite is green" — and the trajectory layer is what stops a lucky-but-broken agent from shipping. Every change from here (prompt, model, tool) gets gated by this suite — the seed of the M4 improvement loop.

Self-grade — the M3 gate is nearly closed

Trace is complete — every model/tool step with I/O + timing; run reconstructable from it alone, decisions explicable
Eval has both levels — outcome and trajectory assertions, green on the good agent
Regression caught — suite goes red on your planted bug; a trajectory assertion catches at least one plant
You can debug from the trace — you found the broken step by reading, not re-running

With this plus Lesson 0013's critic step, the M3 gate (debug from traces alone + eval catches a planted regression) is demonstrated.

Lesson 0013 — the critic/verifier step: add a self-correction stage and measure how many silent wrong answers it kills; then the M3 readiness check.

Tracing & the Planted Regression

1 · Instrument a real trace

2 · Build a small eval set

3 · Plant a regression — the lab

Self-grade — the M3 gate is nearly closed

Next