Own the Feedback Loop

M4 in one move: reliability is engineering, the eval suite is the ground truth, and ownership means the loop runs because you run it.

Your win for this lesson: define SLOs for an agent you actually run, place one human checkpoint by risk, and sketch a full observe → fix → verify cycle. That's operational ownership on paper — the M4 gate. (Prerequisite: Lesson 0003 — you can't own targets you can't measure.)

1 · From M3 to M4

M3 gave you the instruments: traces and evals. M4 points them at production and holds the system to targets. ⚠️ The stage's first intuition: production reliability is an engineering discipline, not a prompting problem — when an agent misses its targets weekly, the fix is fallbacks, budgets, gates, and loops, not another adjective in the prompt.

2 · SLOs: agents get pagers too

An agent in production carries service level objectives like any other service (Google SRE Book, ch. 4 — SLI = what you measure, SLO = the target). The agent-specific four: success rate, latency, cost-per-task, and escalation rate — first-class metrics, dashboarded, not vibes. Below the targets sits the reliability toolkit: graceful degradation, fallbacks, retries with backoff, and circuit breakers for flaky tools.

3 · Cost is a design constraint, not an invoice

Attribute cost per run, per user, per feature — then design against a token budget the way you design against a latency budget. Model routing matches task difficulty to model tier instead of defaulting to the biggest model; caching, context compression, and prompt optimization are ongoing levers, not launch-week chores. Cost, reliability, and quality form one tradeoff surface — you tune the system, not the prompt.

4 · Safety: least privilege, checkpoints by risk, audit everything

Guardrails: input/output validation, injection defense, and tool-permission scoping to least privilege — the OWASP Top 10 for LLM Applications is the standard taxonomy (prompt injection sits at #1). ⚠️ The hard design problem is human-in-the-loop placement: which actions require approval, and how to surface them without killing throughput — checkpoints go where risk is, not everywhere. And auditability: every consequential action traceable to a decision and its context. You live inside a worked example daily — this repo's guarded commit paths and human-approval gates on publish actions are exactly a checkpoint policy placed by risk.

5 · The improvement loop is the job

Observe → evaluate → hypothesize → change → re-evaluate, continuously, with human feedback recalibrating the metrics. Regression suites gate every prompt, model, and tool change before it ships. ⚠️ And the intuition that makes the eval suite existential: the model will change under you — new versions silently shift behavior, so your eval suite is the only stable ground truth you own. Ownership means owning this loop — spotting drift, shipping gated fixes, and proving the fix held — without being told what to fix.

The M4 load-bearing sentence: you run an agentic system the way you run any production system — against SLOs, inside budgets, behind risk-placed human checkpoints — except your only stable ground truth is the eval suite, because the model itself will change under you.

6 · Retrieval quiz — closed book

Answer from memory. Click an option for instant feedback.

1. Your production agent misses its targets weekly. This is primarily a…

Production reliability is an engineering discipline, not a prompting problem — fallbacks, retries, circuit breakers, budgets, gates.

2. A model version bump silently shifts behavior. Your stable ground truth is…

The model will change under you. The eval suite is the one thing you own that says "this still works" — which is why it gates every change.

3. First-class SLO metrics for a production agent:

The agent-specific four: success rate, latency, cost-per-task, escalation rate — measured per run, dashboarded, owned.

4. Where do token budgets belong?

Token budgets are a design constraint — like a latency budget. If cost only shows up on the invoice, governance already failed.

5. The hard part of human-in-the-loop design is…

Checkpoints go where the risk is. Approval on everything is safe and useless; approval on nothing is fast and reckless — placement by risk is the craft.

6. Cost, reliability, and quality are…

One tradeoff surface — you tune the system, not the prompt. Pushing one corner moves the other two; owning the system means owning the whole surface.

7 · Ownership on paper — the gate skill

The M4 signal is owning SLOs and closing improvement loops unprompted. Pick one agent you actually run today (a Vernant pipeline stage, an AFK run, a scheduled job). Write: its four SLOs with targets (or "measure first, then set"), one high-stakes action plus your checkpoint policy for it and its throughput cost, and one full improvement-loop cycle. Reveal the rubric, self-grade, bring it to the next session.

A passing design covers all four:

All four SLOs — success rate, latency, cost-per-task, escalation rate, each with a target or an explicit "measure first" plan (a vibe is not a target)
Checkpoint by risk — one high-stakes action gated by human approval, with the throughput cost named and accepted, not ignored
A complete loop — observe → evaluate → hypothesize → change → re-evaluate, where the re-evaluation runs through the eval/regression suite, not through eyeballing
The gate stated — what regression suite blocks a bad prompt/model/tool change from shipping, and why it (not the vendor) is your ground truth

Next — the map is laid; the proofs remain

Lessons 0001–0004 are the four-milestone mental-model spine of the whole knowledge map. From here, everything is hands-on: the M1 single-tool agent build is the first proof due, then up the ladder — each gate passes on demonstrated capability, not coverage.