Learning / Agentic Operations / Lesson 0018

Agentic Operations · Lesson 0018 · Level 4 · Operational Leadership · Capstone → gate

The Full Improvement Loop

Ownership means owning the loop. Run it once, end to end — that's the last thing to prove.

Your win for this lesson: one complete improvement cycle — catch a live regression, ship a gated fix, and prove your eval suite would have blocked the bad version — the rubric's third Level 4 exercise, and the whole map's final proof. Prerequisites: everything, especially 0014 (evals), 0016 (SLOs), 0017 (gate).

The loop, run once for real

The Level 4 improvement loop is observe → evaluate → hypothesize → change → re-evaluate, run continuously (Lesson 0004). You'll execute one full turn of it — the way you'd run it forever in production.

  1. Observe. Introduce a "live regression" the way reality would: bump the model version, or change a prompt, or swap a tool — something plausibly innocent. Let your 0016 dashboard surface the damage: an SLO drops (success rate falls, or escalation spikes). You find it because you're watching the metrics, not because a user complained.
  2. Evaluate. Reach for the traces (0014), not the prompt. Read the failing runs and locate the faulting step. This is the Level 3 reflex serving the Level 4 loop.
  3. Hypothesize. State what you think broke and why, as a falsifiable claim ("the new prompt made the agent skip the verifier on short inputs").
  4. Change. Ship the fix — through your 0017 guardrails/gate, not around them. The fix is a normal gated change, not a hero patch.
  5. Re-evaluate. Run the eval suite (0014). It must go green again — and, critically, add a test case that reproduces this regression, so the suite now catches this class forever. Prove it: run the new suite against the bad version and watch it go red.
Pass: you closed the loop unassisted — spotted the drift on the dashboard, diagnosed from traces, shipped a gated fix, and left the eval suite strictly stronger than before (it now blocks the bad version you started with). Prove the last part explicitly: the enhanced suite is red on the regression and green on the fix.
The eval suite is your ground truth. ⚠️ The model will change under you — new versions silently shift behavior (Lesson 0004). The one stable thing you own is the suite, and it just got stronger. That's what "owning the loop" means: not shipping the current build, but running the machine that keeps every future build honest.

The Level 4 readiness check — and the whole arc

Bring to your teaching agent: your SLO dashboard for a live agent (0016) and a complete observe→fix→verify cycle you ran independently (this lesson). Those are the literal Level 4 gate — and passing it is the top of the map.

The doc's own signals that you're ready for ownership — the check probes each: Demonstrate these and you've reached Head-of-Agentic-Operations capability: the destination of the whole knowledge map.

The map is complete

Eighteen lessons: four mental-model spines (0001–0004) and fourteen hands-on proofs (0005–0018), one arc per milestone, each gated on demonstrated capability. You started at "an LLM predicts the next token" and end owning the loop that keeps a production agent honest as the model shifts beneath it. There's nothing above this in the rubric — from here it's depth, scale, and the judgment that only real operational reps build. That's the wisdom the method always said comes from practice, not pages.

You've finished the course

Congratulations. This was the final lesson — all 18 lessons, all four level gates: you can explain why "not in context = doesn't exist," you've shipped and broken a real agent, you've justified pattern choices, debugged from traces alone, caught a planted regression, and run the full observe → fix → verify loop with your own SLOs. That's the whole map. From here, the course's habits belong in your real systems — and the course index and reference sheets stay put whenever you want to revisit a rung of the ladder.