Is Your AI Agent Checking Its Work, or Just Describing It?

My publish control told me it was done. It wasn’t.

I found out by accident, in the middle of a conversation about something else. I had been asking Claude Code whether a heavier agent-orchestration framework was worth adopting for blog work. The answer was mostly no, which is what I expected and roughly what I wanted to hear. But the reasoning went through my git history on the way to getting there, and it came back with this:

Every one of those commits is the same failure class. Publish something, watch the site’s metadata drift out of sync with it, ship a follow-up commit to close the gap. Six visible instances of a recurring cleanup that I had never named as a category, because each one had looked like a one-off tidy at the time.

I was talking myself out of adopting more automation when I found it. That detail matters to me more than it probably should. I did not run a scheduled review. I did not have a hunch. I got handed a pattern in my own history while arguing for less machinery, and then I went and checked the site against itself for the first time.

Four articles, drifting in two directions

The first three had fallen off /articles/, which is the hub ... the page a reader lands on when they click “Writing” in my nav. Anyone who browsed my archive that way saw a collection that stopped in early July. Three of the most recent things I had written were not on it.

The fourth one jumped out at me, because it drifted the opposite way. It was on the hub and missing from the homepage, and it was missing from llms.txt, the file AI crawlers read to understand what a site contains. Same site, same protocol, same month, failing in the reverse direction.

There is no consistent pattern to watch for. That is the part that killed the option of just being more careful.

I already had a control

This is the part where the story would be simpler if I had been publishing by hand.

In June I wrote an article about how I don’t publish articles ... I release them through an AI control.¹ It was a two-agent loop, and I was pleased with it. Claude Code executed the dependency updates for a new post, produced a structured log, and then prompted me to paste that log into Claude chat, where it got validated against the authoritative protocol stored in my Open Brain wiki. Executor on one side, validator on the other.

Between June 14 and August 2 I published five more articles through it. Every run produced a clean log. Every [DONE] line in every one of those logs was accurate about what the agent believed it had done.

So the question in front of me was not why I had no control. I had a control, it had a second reviewer, and it passed six times in a row while three articles quietly walked off the hub. If you are about to tell me that I discovered unit tests, hold that for a minute, because the review layer is the interesting part...

The step did not exist to be skipped

The first failure is the simpler one, and it is the kind of thing that shows up in audit work constantly.

My protocol covered sitemap.xml, llms.txt, and the homepage Writing section. It never named articles/index.html.

So those three articles were published correctly. Not sloppily, not in a rush, not with a step skipped because it was late and I wanted to hit publish. They were published in full compliance with the protocol as written, and the protocol was wrong. The step was not skipped. The step did not exist to be skipped.

No amount of diligence in execution finds a missing requirement. Only something that checks the outcome does.

If you test controls for a living you have already written this finding about somebody else. A control matrix that never maps a risk produces immaculate testing results for the risks it does map, and the immaculate results are what makes it dangerous ... you come away more confident than when you started.

A control whose evidence was testimony

The second failure is the one worth the article.

My publish logs ended with lines like this:

Then I pasted that log into Claude chat and asked it to validate the run against my wiki protocol. Which it did, carefully, every time.

Neither model ever opened a file.

One agent asserted it had done the work. A second agent read the assertion and confirmed the assertion was well-formed, complete, and consistent with the protocol. Both of those things were true. Neither of them was evidence. A review whose only input is the executor’s account of its own work will pass forever, because an account is always consistent with itself ... that is what makes it an account.

I had built a control whose evidence was testimony.

The audit parallel is exact and it is not flattering. Taking management’s representation as evidence without independently verifying it is a finding I have written about other people, in reports. I did it to myself in my own publishing pipeline and then wrote a well-received article about the architecture.

Which brings me to the closing line of that article:

I still think that is right. It is also not enough, and I had the proof sitting in my own repo while I wrote it. I had the separation. Both halves were language models, and both of them were reading the same description. Segregation of duties assumes the second party can see something the first party cannot. Mine couldn’t. It could only see what the first one had said.

Moving the control plane out of the prompt

The fix came from a premise I picked up from IndyDevDan, who works this problem publicly on YouTube and shared the repo for his Super Simple Software Factory.² The premise is one line: move the control plane out of the prompt and into Python.

I looked at his framework and did not adopt it. It is more machinery than a static HTML site with no build step needs, and one of his own stated principles is to listen to feedback and then think for yourself, so I took the idea and left the package. The idea is the hinge of everything below. Sequencing, the pass/fail decision, and verification stop being things an agent is asked to do in a prompt and become things deterministic code does. The agent keeps the work that needs reading and judgment.

What that turned into is scripts/check_site.py. Plain Python, standard library only, no dependencies, no virtualenv, resolving its paths from its own location so it runs from any directory on either of the two machines I publish from. A little over two hundred lines.

The design principle that carries it: the filesystem is the source of truth. The script does not read a list of what should exist. It walks the repo, finds every article page that does exist on disk, and then proves that each one appears on every metadata surface with the tags it needs. Right now that is fifteen pages. When it is sixteen, the script will know without being told, because nobody has to remember to update it. There is nothing in it to maintain and nothing in it to go stale.

It runs five groups of checks:

Exit 0 clean, exit 1 with every violation named. The publish skill can no longer write [DONE] against a step the checker did not confirm, which is the whole point of building it.

The first run was mostly my fault

First execution reported nineteen violations and seven warnings. Roughly half of them were the checker being wrong, not the site.

Two are worth showing, because they are the ordinary texture of standing up a new test and nobody writes about them.

The footer check. I told it to look for class="connect-footer". My archive turned out to have four generations of footer markup ... connect-footer, article-footer, site-footer, and author-footer ... plus three early pages that style the identical block inline with no class on it at all. Every one of those pages was fine. My check was matching the markup of the week instead of the thing I actually cared about, which is a closing block with a way to reach me. I rewrote it to look for the block and the LinkedIn link, version-agnostic.

Reference forms. My surfaces link to articles four different ways: absolute URL, root-relative, relative to the site root, and relative to /articles/. My first pass knew two of them and reported perfectly healthy pages as missing.

Nineteen violations, then nine, then two, then zero.

A new control’s first run is mostly noise, and separating real findings from your own bad test logic is the actual work. Anyone who has stood up a monitoring rule or written a new audit test knows the feeling. Treat that first output as gospel and you will spend a week chasing ghosts. Get tired of it and start loosening thresholds and you end up somewhere worse ... a control that passes because it no longer looks.

Two of the nineteen were not defects at all. They were policy questions wearing a violation’s clothes: should my three earliest pages, which live at the site root rather than under /articles/, appear on the hub? That needed a decision from me, not a fix from the script. The answer was yes, and I migrated them.

And four of my titles run past sixty characters on purpose, because the full phrasing does more for the piece than the search snippet loses. Those four are recorded in an ACCEPTED_LONG_TITLES set in the script, dated and reasoned, and a fifth long title still warns. Recording an accepted exception and silencing a control produce the same quiet output on the next run. Only one of them survives someone asking why.

Three records, all wrong, and the disk

Here is the thing that made me go hmmm.

My June protocol contained an AI “hand-maintained” Page Inventory table. Every URL, with checkmarks for sitemap, llms.txt, and the homepage card. A coverage table. The control for exactly the drift I had just found.

It had drifted. Last refreshed June 27. It had never listed three of the articles. It carried three separate notes reading “reconcile on a future lint pass,” and no future lint pass had ever come.

A hand-maintained coverage table cannot be the control for coverage drift. It drifts the same way and more quietly, because a stale checkmark looks exactly like a fresh one.

So I retired it. The checker derives that same coverage from disk on every run, which removes the reconciliation work rather than scheduling it.

Then I went one layer out and found it again. My blog pipeline tracker, the wiki page where I count published posts against a goal, was missing two published articles and reporting the wrong live count ... partly because a 2016 piece hosted on LinkedIn was logged in a way that read like a page on my own site.

And there was very nearly a third. I publish from two machines, and while rebuilding this I wanted to keep a copy of the protocol somewhere convenient, like a doc in Drive. That would have been a second source of truth for the same information, which is the same disease with better ergonomics. Git is the sync mechanism now. The wiki documents where the files live in the repo instead of holding copies of them, and the local skill carries a fallback plus a divergence check against the wiki ... which means changing the local copy first fires that check on both machines, correctly, so the wiki had to move first. A control catching me doing things in the wrong order is a small, happy piece of evidence that the design works.

Three layers of records about the same fifteen articles. All three had drifted. The only one that was right was the disk.

What each half is for

The division that came out of this is the thing I would keep if I had to throw the rest away.

The Python checker mechanically provides facts. Which files contain which strings, whether a tag is present, whether a link resolves, whether an exit code is zero. Questions with one right answer, checked identically every time, by something with no stake in the result and no ability to be persuaded.

The model owns judgment. Does this excerpt represent what the piece actually says. Is this description honest. Is the framing right. Does the meta description promise something the article doesn’t deliver.

Before, I had one model doing both, and it was doing the first job badly enough to make the second job worthless.

The checker cannot tell me whether an excerpt represents a piece. It will confirm that a description exists, that it is under a hundred and sixty characters, and that it matches the string in llms.txt exactly, and it has no opinion whatsoever about whether the sentence is true. That is still my job and the model’s. I want to be plain about it, because a verifier that passes clean is a good feeling and good feelings are how you stop reading your own work.

The auditability of it

Every post in this series ends with the control I added, so here it is stated as a control rather than a story.

The evidence changed form. What used to be a representation ... an agent’s account of its run ... is now a verifier execution with an exit code, and its real output goes into the publish log. Exceptions are recorded rather than silenced: the long-title allowlist is dated, reasoned, and still warns on anything new. The coverage record derives from the system instead of sitting alongside it, so there is no reconciliation left to schedule and nothing to forget. And the segregation of duties is finally worth the name, in a narrow way ... the thing that does the work no longer attests to the work.

None of that makes the pipeline correct. It makes the pipeline checkable, which is a smaller claim and the only one I can support.

What I actually want

I have spent a lot of time lately close to systems built to a particular standard: narrow and well-defined in scope, outputs a person can verify, data lineage you can document, and logic a human can interrogate and explain out loud without hedging. Not systems that are trusted because they sound right. Systems whose output arrives with something underneath it.

check_site.py is about as small an example of that as exists. Two hundred lines, no dependencies, one exit code. It has all four of those properties, which is exactly why I trust it and why I no longer trust a log line that says [DONE]. The difference between those two is not intelligence. The agent is far more capable than the script. The script opens the file.

What I don’t have a good general answer for is how you build that at a scale where nobody can read all of it. My site is fifteen pages and a static host, and the verifier fits on a screen. If your agents are touching customer records, ledger entries, or anything a regulator will eventually ask you about, what does the equivalent look like ... and who checks that it ran? I have been circling that question for a while and I expect to be writing about it for a while longer.

Start with the small version. Pick the one claim your automation makes that you have been taking on faith, and go write the twenty lines that open the file.