Ten runs, one lever

I re-ran the same AI code review ten ways to find out what actually makes a second reviewer worth paying for. Then the reviewers I built found three errors in it — one in how I scored it, one in a flag that did nothing, and one in a cost figure that was several times too low, then again in my correction to it — and took two of my three conclusions with them. Every correction is below, in order.

In my home fleet, every pull request gets an adversarial model review before a human merges it. The rule is coder ≠ reviewer ≠ merger. It works well enough that I wrote about it the last time, when one PR went nine rounds before I ran out of budget.

But it left an obvious question unanswered. If one review is good, is two better — and if so, which two? Run the same model twice? Turn the reasoning effort up? Or pay for a second model from a different family? Those cost roughly the same and I had no idea which one bought anything.

So I measured it. The nice thing about a review pipeline is that it is trivially repeatable: the input is a diff and a tree, both frozen in git, and the reviewer has no memory. I picked a review that had been long and genuinely fruitful, rolled the branch back to the commit it had run against, and re-ran it ten times under varied conditions.

The setup

The subject is a real PR: the initial extraction of a secret-guard — a pre-tool hook that blocks an agent from reading credential files — plus a transcript leak-watcher and a telemetry collector. About 90 KB of diff against 87 KB of surrounding source, which the dispatcher packs into a prompt of 44,285 input tokens. Identical for every run.

Three variables, one at a time:

• Sampling — the same model, same effort, run three more times with a fresh context each time. If a missed finding is just an unlucky roll of the dice, re-rolling should recover it.
• Effort — the same model at a higher reasoning-effort setting, and the previous generation of the same family at high effort, in case the model version was the confound rather than the effort.
• Family — a different model family entirely, at high effort.

Ground truth is the union of the defects found across the whole set, checked by hand against the code. I want to be precise about what that means, because it is the honest limitation of the whole exercise: it measures recall against what some run found, not recall against every defect in the PR. A bug that all ten runs missed is invisible here. So the result below is a floor on the disagreement between models, not a measurement of their absolute skill.

What came back

Both right-hand columns are superseded. The "prompt said" column was labelled "effort" until I discovered the setting was never applied, and the cost column is the single-prompt estimate, which turned out to be several times low — the measured figures are further down. They are left here because the piece is partly about how confident wrong numbers look.

Finding counts are a soft metric — reviewers slice the same defect differently, and one run's single MEDIUM is another's three LOWs — so read the column as texture, not as a score. What matters is which defects appeared where.¹

Sampling: what I thought I found

Four independent runs of the same model at the same effort, each with a fresh context. They did not agree on how to carve up or rank the findings, but the substance came back every time: the glob and quoting forms that defeat the path matcher, the .env.local family the regex misses, the directory-scoped read that walks straight past a file-name rule, the line-splitting bug that silently drops transcript lines.

That was the hypothesis I believed going in, and I wrote it up as the finding: whatever a model is blind to, it is consistently blind to. A much less comfortable property than randomness, because randomness you can fix by paying twice.

It was wrong, and the error was mine. Keep reading.

Effort: a measurement that never happened

This is the part I did not expect. The same model at high effort took 435 seconds — a third longer than its own median at medium — and came back with three findings where the four medium runs averaged ten. It found the headline bypass and two low-severity offset bugs, wrote them up beautifully, and stopped.

Notably, it did not report the transcript line-splitting bug that all four medium runs found. That one is a genuine silent-failure path: a tool result containing a Unicode line separator gets split into fragments, neither fragment parses as JSON, the whole result is skipped, and any credential inside it is never scanned. Higher effort on the same model missed a real defect that lower effort caught.

I ran the previous generation of the same family at high effort as a control, in case the version was the confound. Same shape: four and six findings, both missing the splitting bug. So the effort dial did not trade breadth for depth in any way I could see — it traded breadth for prose. Longer reasoning, fewer places looked at.

All of which would be interesting if the dial had been connected. It wasn't.

The reviewer that read this article as a pull request checked my dispatcher against my claims and found that --effort never set a reasoning-effort parameter. On the CLI path every arm used, it substitutes a single sentence of prompt text — "Review depth for this run: high" — and the code says so in a comment I wrote: "Effort rides in the prompt rather than a flag: the CLI has no --effort." The real knob exists on a different code path that none of the runs took.

So the arms above differ by prompt wording, not by effort. "Effort doesn't buy recall" is not a weak finding here; it is not a finding. I withdraw it.

The part I find least comfortable is that this was already written down. A note from an earlier session says, in as many words, that the flag is prompt text only. I read it at the start of the day and then built an experiment, a policy and this article on top of the opposite assumption. Knowing something, and having recorded it, turns out not to be the same as checking it.

Family: the whole effect

The Opus run found the worst bug in the PR — and, I wrote at the time, no Fable run found it at any effort across seven attempts. Hold that second clause loosely; the next section is about how it turned out to be false.

The guard's README makes an absolute promise: "There is no in-band override. Nothing a session puts in a command bypasses a match." Every Fable run read that sentence and went hunting for commands that bypass a match — globs, quotes, backslashes, directory paths. They found several. Good work, all of it real.

Opus asked a different question: what if you don't bypass the match, and instead change what matches? The rules live in a config file. The config file is covered by no rule. And the hook's documented matcher is Bash|Read|Grep — the file editing tools are not in it. So an agent can simply rewrite the rules and then do the thing. Opus verified the whole chain against the shipped binary:

The fail-closed handling was careful and correct — it catches a mangled config. But a config with one dummy rule of each kind parses as healthy, logs nothing, alerts nothing, and enforces nothing. The guard reports fine and does not guard.

And the effect runs both ways, which is the part that survives the correction below. Opus missed the line-splitting bug that all four Fable-at-medium runs caught. It touched the same function — it filed the wrong-line-number bug two lines away — and did not see the silent drop.

So neither model is the better reviewer. They have different blind spots. That is what I published.

Where it fell over

The pipeline that produced this experiment also reviews the pull requests that document it. When the write-up went through as a change to the routing table, a reviewer read the scoring table against the eight saved review files and said, in effect: your arm B3 is a hit, not a miss.

It checked out. B3 — one of the four Fable-at-medium runs — says a session "can rewrite config/secret-patterns.conf (or bin/secret-guard itself, which resolves the conf relative to its own location) and the next tool call runs unguarded." That is the defect. It even anticipates, by two rounds, the executable version of the hole that I later credited to Opus alone.

I had read it as a documentation complaint. Two other arms quote the same README sentence while reporting a genuinely different bug — the glob bypass — and those two are real misses, which is what made the misreading easy. It isn't much of an excuse. I had already thrown out one automated scoring pass for being unreliable and switched to reading the texts; I then read this one too fast.

What survives

Rescored, the picture inverts in the least convenient direction:

• Sampling works. B1, B2 and B3 are three draws at identical settings, and only B3 found the config hole. That is variance, not a shared blind spot — and it makes re-running the cheap thing the best-supported intervention in the whole experiment. My original conclusion said the exact opposite.
• Effort was never tested. The dial wasn't connected — see above. This was the conclusion I thought had survived intact, and it turned out to be the one that had never been measured at all.
• Family diversity is one observation each way. Opus found the config hole — so did one Fable run. Every Fable-at-medium run found the U+2028 drop and Opus did not. Consistent with the two being differently blind; nowhere near enough to establish it, and certainly not the "nine runs missed it, one found it" I led with.

The cost figures hold: the second opinion really is the cheaper one. But the reason I gave for buying it is now a reasonable prior rather than a measured result, and those are different things to publish.

Counting honestly, this experiment had three conclusions and lost two of them to two different kinds of error — one a misread of my own data, one a flag that did nothing. What is left is a single sentence: two model families, given the same prompt, found different defects in both directions, and the cheaper family found the more severe one. I would still pair them on that. I would not have written a policy on it.

I've left the original claims above rather than quietly editing them, because the shape of the mistake is the most useful thing here. An experiment that re-runs one review ten times is a machine for producing a confident number, and the number was wrong at the first step — not the arithmetic, not the method, but a judgement call about whether a paragraph counted as a finding. Every conclusion downstream inherited it. If you build something like this, the scoring is the experiment; the runs are just input.

The cost, which surprised me

I assumed the diverse second opinion would be the expensive one. It is the cheap one.

This section has now been wrong twice: once as an estimate, and once as the correction to that estimate. It carries all three versions, because a piece about confident wrong numbers should not quietly arrive at the right one.

What I estimated. At list prices — Opus 5 at $5/$25 per million tokens in/out, Fable 5 at $10/$50 — a 44,285-token review priced out at $0.30 on Opus against $0.51–$0.54 on Fable.

Why that was wrong. It priced a single prompt. These reviews are not single prompts: the reviewer has tools and uses them, so a run is 8 to 17 turns, and the payload is written to cache on the first turn and re-read on every turn after. A reviewer on the pull request describing this experiment said exactly that, and I recorded the finding and published the numbers anyway.

What I corrected it to, which was also wrong. I queried the fleet's telemetry, published a per-model table, and called the error "roughly twenty times". Two things were wrong with that. The per-model filter matched model names by substring, so claude-fable-5 also caught every claude-fable-5-1 session and that row was a blend of both. And the table's rows summed to 163 while its own headline said 120 — a contradiction printed in the same paragraph, which I did not notice and two reviewers did.

What it actually costs, with exact model matching, from 124 review sessions:

And the multiplier was not twenty. Per review it is 4.7× to 7.0×. The twenty-fold figure came from comparing a whole-night total against an estimate that had also assumed the wrong number of reviews — so it multiplied two separate errors together and reported the product as one. That is a subtler mistake than the original and I repeated it in four places before it was caught.

The conclusion has survived all three versions unchanged, which is the only reason I still believe it. Opus 5 costs less than either Fable — $2.07 against $2.41 — and earns that while running twice as many turns and writing 80% more output. It is not cheaper because it does less.

Note what I am not claiming, because I claimed it in the previous draft and a reviewer caught the contradiction in my own table: Opus 5 is not the cheapest row. Opus 4.8 is, at $1.99 across five reviews. Five is too few to conclude anything from, and it is not the model the routing rule names — but writing "the cheapest reviewer in the set" directly above a cheaper row is the kind of error that makes a reader stop trusting the rest of the page, correctly.

The pairing costs about $4.48 per tier-3 review, not the $0.83 I first published. So the ranking that made the decision is unchanged, and every absolute number attached to it has now been wrong twice.

Who was running the experiment

Worth stating, since it is the kind of thing that ought to be disclosed rather than discovered. The orchestrator driving these runs — the session that rolled the branch back, launched the arms, read the outputs, and drafted this piece — is itself a frontier model, in this case Opus, invoked headless. It is the same family as arm D.

I don't think that biased the result, because the scoring is not a judgement call: the pivotal claims are "this arm's text contains a finding about the writable config" and "this arm's text does not," which anyone can check against the eight saved review files. But an orchestrator evaluating its own family is exactly the setup where you should want the raw artifacts, so they are kept.

It did not go cleanly, either. An early scoring pass tried to match findings by keyword and produced two tables that contradicted each other — one arm scored a hit on a substring that appeared in a "things I checked and found sound" section, and another was scored a miss because it had described the same defect in different words. The keyword pass got thrown out and the eight reviews were read. Worth knowing if you plan to automate this: the cheap way to score model outputs against each other does not work, and it fails silently, in both directions.

What I changed

The policy now requires two adversarial passes from different model families at tier 3 — credentials, auth, crypto, trust boundaries, privacy — and one everywhere else.

The scope limit is deliberate and it is the part I would push back on if someone else wrote this. What I measured is recall. I did not measure whether a second pass pays for itself at lower tiers, where a missed defect gets caught later by a test, a user, or the next person to touch the file. Those are different questions and the experiment only answers the first. Extending the rule to every tier would be exactly the sort of uncited assignment the routing table exists to prevent.

Three things I would take somewhere else:

• Run it again before you buy anything cleverer. Three identical draws disagreed about the worst defect in the PR. Resampling is the cheapest intervention available and it is the one this experiment actually supports.
• Diversity is probably worth it, on a prior rather than a proof. A different family found something, and costs less than the first reviewer. That is a good enough reason to pair them. It is not the same as having measured it, and I published the stronger version first.
• Check the price before you assume the diverse option is the premium one — and then check it against telemetry rather than arithmetic. The second opinion really is cheaper, by 14% rather than the 43% I first calculated, and I only know that because the fleet was recording per-turn token usage the whole time and nobody had looked.

And the caveat now has to be stronger than the one I first wrote. This is a single task, on a single PR, with n=1 per direction on the family question and a scoring pass that had to be corrected once. I would not bet on the numbers. What I would take from it is the process lesson rather than the result: the reviewer that caught my error was doing exactly the job I built it for, on the document describing itself.