Six things I got wrong

A governance practice is only worth as much as its correction record. Here is mine — what my own review pipeline caught me publishing, and the gap list underneath it.

The fastest way to describe a private lab is architecture: a model gateway, an identity plane, a secrets vault, a review pipeline where one model writes code, a different model reviews it, and a human merges. That description is also useless, because it is exactly what a weekend project produces. Architecture is the cheap part.

What is expensive is elapsed time under scrutiny. So rather than describe the design, here is what it has caught — including, mostly, me.

The correction record

Every item below was found by the adversarial review pipeline described further down, and corrected with a changelog entry saying what changed and why. All six are from a single 48-hour window.

Three of them were public and three were not, and the distinction matters more than the count. The first three shipped to this site, so you can check them — though "corrected" is itself a claim with a state: at the time of writing, the cost figure's correction is a pending change rather than a live one, which is exactly the gap between fixing something in a repository and fixing it where anyone can see. The last three were commits in a private repository, caught before anyone outside saw them — and for two, the correction is still an unmerged pull request as I write this. I have kept them in the list because they are the more instructive failures, but a reader can only falsify the first half, and a page arguing for falsifiability should say which half that is.

I am leading with this because it is the only part of a governance practice that cannot be assembled in an afternoon. Anyone can write a policy. A policy with a record of being wrong, caught, and corrected has been running — and for the published half the record is falsifiable, which a description of architecture never is.

The mechanism that caught them is the one the policy describes: every change gets an adversarial model review before a human merges it, and at tier 3 — credentials, auth, privacy — two reviews, from different model families. The two-family rule is scoped to tier 3 deliberately; extending it everywhere would be an assignment no measurement supports. In one night that pipeline ran 124 reviews on 17 pull requests, against telemetry covering 1,130 sessions and 68,181 turns. Several of the corrections were defects inside fixes the reviewer had itself requested, which is the part I found least comfortable and most instructive.

And the gaps, since that is the other half

Two external frames were the obvious way to find what I had missed: ISO/IEC 42001:2023, the AI management system standard, and Anthropic's architect competency blueprint. They were built for different purposes and agree about almost nothing — which is why the places they both point at are worth taking seriously.

Two things this is not

Not a conformance claim. Conformance to ISO/IEC 42001 is a determination an accredited certification body makes after an audit. Nothing here has been audited, and certification is not the path — not on cost, but because certification bodies certify organisations with a management system, and there is no entity here to certify. The standard's structure is free of that constraint and is the part that produces findings.

Not a score. I am publishing the gap list rather than a set of green ticks, because the gap list is the part that is actually informative. Anyone can write "meets control A.6". Fewer people will tell you which six things they know are missing and what each would cost to fix.

The gaps

The two that are actually interesting

Gap 5 made the assessment worse, on purpose

The boundary question is whether the management system covers only the AI I use to build software, or also the AI features inside things I ship.

The narrow answer is much easier to conform to, because under it the only affected party is me. Several controls about data governance and informing interested parties simply do not bind, and the map looks tidy.

I chose the wide boundary, and three ratings went down as a direct result: impact assessment — which is gap 1 above — plus two Annex A controls that are not in the table because they had nothing to rate until the boundary moved. Data for AI systems and information for interested parties both acquired a subject the practice does not currently serve.

The second population is the one that makes this concrete, and I missed it in the first draft. The product places autonomous phone calls to third parties. Those people never chose it, have no relationship with it, and cannot opt out — and they are inside the boundary too. A disclosure designed to reduce hangups is not the same artifact as one designed to inform someone, and only one of those is what the standard asks for.

That is the correct direction for a scope question to resolve if it is being answered honestly. A boundary drawn so that controls become inapplicable answers nothing — it just moves the problem outside the frame and declares victory. The tell for a self-assessment worth reading is whether any decision in it made the results worse.

Gap 4 was excluded for a reason that stopped being true

Bias and fairness were left out of the written policy deliberately rather than by oversight, on the argument that generating and reviewing code makes no consequential decisions about people. Under a narrow boundary that holds.

It does not survive the wide one, and the estimate went from half a day to days when the boundary moved. Half a day was the cost of a policy paragraph. The real work is identifying who could be disadvantaged, deciding what would count as evidence, and testing for it — and none of it is done.

The document says so rather than adding a section that gestures at fairness. A thin treatment is worse than an admitted gap, because it lets a reader believe the question was asked.

What the practice does have

For balance, since a document organised around holes loses this. Three controls that exist and have been observed working, which is a deliberately higher bar than "specified":

• Approval bound to a content hash. Merging one change into another's base voided two standing approvals and forced a re-review — which then found a live defect in a credential-matching rule that would otherwise have shipped.
• Capability events as a policy trigger. Observing a new model capability obliges a policy review. Exercised once: a discovery about cross-session state produced a new required field in the same revision.
• "Not real until observed to fire." A safety feature may not be relied on until it has been seen to work at least once. This one keeps earning its place — it recently caught a staleness warning that was being overwritten one line after it was set, and a safety check whose prescribed test was structurally blind to the very incident it had been written from.

An earlier draft of that list had five entries. Two were things the practice specifies rather than things it does, and a reviewer caught the difference. They have been moved to a separate list, because a section boasting about extras is the last place an overstatement should be allowed to survive.

Why any of this is worth publishing

The alternative is a page of green ticks, which tells a reader nothing they could check. But the real reason is narrower: the interesting property of a self-assessment is not its result, it is whether the author was willing to end up worse off for having done it honestly.

Two decisions on this page cost me something. The scope boundary went wide, which lowered three ratings. Bias and fairness came into scope, which turned a closed question into days of unfinished work. Either could have gone the other way and produced a tidier document.

And the six corrections at the top are the same test applied to the practice rather than the paperwork. A pipeline that never contradicts its operator is not a pipeline, it is a rubber stamp. The experiment where two of three conclusions did not survive contact with their own data is written up separately, and the nine-round review that produced the pipeline is here.