My mustang's paperwork and my agent fleet's records shared one fault: nobody had ever set them side by side. This month I built the thing that does.
Earlier this summer I went looking for where our mustang Red was born wild. His federal paperwork said one thing; my own notes said another. Neither was lying — they were written at different times by different hands, and nobody had ever set them side by side.
So I set them side by side. I read the freezemark under his mane, pulled his full history off the federal site, and worked through it line by line — it took quite a while after getting the brand read. It settled it: Red is an Oregon horse, gathered in the 2021 drought, and my Nevada guess came from where he entered the system, not where he was born. The whole story turned out bigger than either record I had; it's here if you want it.
My agent fleet — the AI agents that run the estate's jobs — had the same problem everywhere. This month I built the thing that fixes it, and it's the most useful thing my agents and I have made in a while.
Every monitor was green
The fleet keeps its records the way the government keeps mustang paperwork: one kind of fact per file. The schedule knows which jobs exist; the run ledger knows what actually ran; the vault knows which machines are real; one Python module knows which AI providers I trust with what. Each record has its own monitor, and every monitor was green.
Nobody was reading them together. For weeks the run ledger carried one of my machines under two names, as if it were two — and every dashboard read one record, so every dashboard was happy.
An ontology, minus the graph
An ontology here is not a graph. It's a short list of the kinds of things in the estate, one file or module allowed to be the authority for each, and the relations that must hold between them. Think of balancing a checkbook against the bank statement: neither record is wrong, and the whole point is the lines where they don't match.
Twelve kinds so far — principle, decision, provider, job, host, box provider, resource, purpose, goal, skill, task, switch — and twenty-one constraints. Most are one of four shapes:
- Referential. Every X named in A exists in B. Every decision cited in the doctrine is a file in the decisions directory.
- Triad. Three authorities must agree. A job is declared in the schedule, registered with the monitor, and observed in the run ledger; a job missing from any one is a finding that names which.
- Allocation. Every instance carries a required edge. A cloud resource that names no purpose gets listed as unallocated against the bill.
- Uniqueness. Exactly one home per fact. A hand-kept list of hosts is checked against the vault directory it copies.
The ontology holds no facts of its own. It projects every host, job, provider and switch at run time from the file that already owns it; hand-edit a fact in the ontology and there is nowhere to put it.
Fifty-six findings
The first pass found fifty-six disagreements. An existing monitor already reported ten of the sixteen that survived triage, and we left those alone on purpose — the value here is the findings nothing else can see. Two of the new ones were the ontology's own bugs, which is why the method now says a finding is a hypothesis until you open the file it names.
The rest were real, and a little humbling:
- In August we built one module to be the fleet's only Telegram sender, and its own docstring said the copies were gone. Twenty-four scripts were still calling the API directly. Seventeen go through the module now; the other seven carry a comment at the call site saying why they can't — and a marker without a reason is itself a finding.
- An hourly offline test suite was rehearsing with fixture costs, and the fixtures were leaking into the real spend ledger. A month that cost about sixteen dollars read as eighty-five. The only thing that had noticed was my monthly budget goal: one number, slow, unable to name the job.
What it gets me
Findings no single monitor could produce, because they live between records rather than inside one. A short list I trust, with what's already covered elsewhere marked as such, so it never becomes a second alarm. And a model small enough to read in a sitting and cheap to keep, because it stores nothing. The part I didn't expect: it travels. The engine knows nothing about my ranch — the kinds, the authorities and the constraints are one binding file per estate, and we wrote down the method for authoring it. Any estate where several systems each report themselves healthy is a candidate: a cloud account, a client's inventory before a migration. I've already started a second binding.
The word "done"
"Done" is the most expensive word an agent says. Hand one a procedure with real order to it — bringing a new model into the fleet, say: eight phases, the trust question first and the decision record last. It comes back finished, and it usually is. Sometimes a phase in the middle got a sentence instead of the work, and the only record of that is the agent's own word. Nobody lied; the model's picture of the run drifted from the run.
That's Red's paperwork again, except one of the two records was never written down.
I asked three models, blind, what my setup was missing. All three named the same thing: it can't refuse "done." So we built the thing that can. A runbook is still a Markdown file — the written procedure itself — but a run of it is now a record kept outside the agent: which phase is next, what passed by a command, what passed by a person saying so in writing. The agent doesn't get to write that record; it gets to ask.
The first version had a hole, and the agent that built it wrote the hole up as an open question: an agent had to choose to open a run. I read that back and called it what it was, the feature failing. A gate the model can decline to walk through is a gate propped open with a rock — it stops nothing.
The second version doesn't wait to be asked. At the end of every turn, on every lane I run — Claude, Grok, GPT and Gemini alike — the workspace looks at two things: did this turn change any of the files that procedure owns, and did the run move. Change without movement, or change with no run at all, and it refuses the turn.
The evening we turned it on, my agent gave the same small job to four panes, one per model: open a run, touch a governed file, stop. Four refusals, each naming the first phase.
Red's two records are reconciled now. My fleet's records had it worse in one way — every one of them was green, and green next to green tells you nothing. The ontology reads them against each other; the runbook reads the agent's word against the run. Same move every time: find the second record, write down what it says, compare. Nobody is lying; a record nobody checks just drifts, whether a model wrote it or a government office did.
When the transcript says finished and the run says phase 3, the pane sits parked, and the run is the record that's right.
Red's story is settled, and the way it settled is the whole month in one move: a third record, read line by line against the two I already had. My hand notes and the adoption paperwork weren't wrong — they were partial, written at different times by different hands, and nobody had ever set them side by side. Once somebody did, the whole story came out.
The fleet had the same fault, just spread across more files: every record honest, every monitor green, and nothing checked against anything. The fix was the same move, repeated — give each claim a second record. The ontology pairs the estate's files with each other; the runbook pairs the agent's word with the run. Fifty-six findings on the first pass, four refusals on the first evening, and a "done" that finally has something to be checked against.
What's left is small: the first real onboarding through the runbook, and the October call on which constraints go daily. Red stands in the corral with his whole story on file now; the fleet's records are starting to have that too. Nobody was lying — the records just hadn't been introduced.