Two AI reviewers, by the numbers
Every pull request in the lab gets reviewed by two Claude models before it can merge. Seven weeks of that, counted. Snapshot as of 25 October 2026.
Nine rounds with a machine reviewer is what one review loop feels like from the inside. This is the aggregate.
The setup
Agents write the code. Two reviewer accounts read every change, each driven by a configured model, and either one can withhold approval. A finding is rated HIGH, MEDIUM or LOW. A HIGH is meant to block the merge; MEDIUMs and LOWs are filed as follow-up issues and the change ships. For most of this period the two reviewers were Claude Fable 5.1 and Claude Opus 5.5.
- 2,308 reviews on 1,279 pull requests across 36 repositories, 8 September – 25 October
- 386 HIGH findings; 206 pull requests drew at least one
- Each reviewer approved about 9 in 10 pull requests on its first review of them
Does the loop work?
Of the 206 pull requests that drew a HIGH:
- 147 (71%) reached a review with no HIGH standing, 109 of them after a single round of fixes
- 43 were closed without merging, so the flagged code never shipped
- 10 are still open
- 6 merged with a HIGH still standing. I audited five by hand: two were real bugs that reached the main branch and were fixed by later changes, and three were rule or documentation findings. The path that let them through, merging over one reviewer's objection while the other approves, is now an open issue.
On the pull requests that went three rounds or more, a HIGH was present in 93% of first rounds, 100% of second rounds and 29% of third rounds. MEDIUMs held steady at about 2.7 a round, which is expected: they never block, and each round reviews the fix as new code.
The two models
| Fable 5.1 | Opus 5.5 | |
|---|---|---|
| Reviews | 1,402 | 848 |
| Reviews with a HIGH | 12.9% | 18.9% |
| Approved on the first round | 90.2% | 88.3% |
| Median time per review | 5.2 min | 1.9 min |
| Median tokens read per review | 664k | 359k |
| Mean cost per review, list price | $2.53 | $0.64 |
| HIGHs that were rule findings, not bugs | 10% | 30% |
Costs are list-price equivalents computed from each review's own token counts. The reviews actually ran on a subscription, so the real cost is quota.
Same change, two opinions
On the 678 commits both models reviewed, Fable raised a HIGH on 78 and Opus on 116. They agreed on 57.
That looks like a clear win for Opus. It mostly isn't. I took all 80 commits where only one model raised a HIGH and had a third model, one that was neither reviewer, compare each HIGH against everything the other reviewer said about the same commit. Then I checked the survivors against the code.
| Only Opus rated it HIGH | Only Fable rated it HIGH | |
|---|---|---|
| HIGHs | 62 | 21 |
| Other model reported the same defect, at lower severity | 36 | 11 |
| Other model reported a related, narrower issue | 12 | 0 |
| Other model did not mention it | 14 | 10 |
| …of which, real defects confirmed in the code | 10 | 7 |
| A separate cut, across all of them | ||
| Rule or documentation findings, not defects | 17 of 62 | 5 of 21 |
Most of the gap is calibration: Opus calls HIGH on things Fable reports as a MEDIUM, and it enforces process rules, like citing where a cross-repository call comes from, much more strictly. What remains is close. Opus found 10 real defects Fable never mentioned; Fable found 7 that Opus never mentioned. None of the 17 reached the main branch: each was fixed before merge, or its pull request was closed and the work re-landed without it. The two real bugs that did reach main, in the audit above, are not among the 17. One merged before Opus was reviewing, so only one model ever saw it. For the other, the second model reported the same defect at lower severity. In both cases one reviewer rated it HIGH and the merge went ahead anyway.
Some of them, described by what they would have done:
- Any holder of a machine credential could mint the owner's real-time voice token. Opus only.
- Asking to extend a calendar event would have deleted the event and created nothing in its place. Opus only.
- A crisis-safety flow stopped offering the crisis line once it had ended a conversation, even if the person then disclosed self-harm. Opus only.
- Any subscriber could submit work against a product they had not been granted. Fable only.
- A build step used a shell feature that the build machine's older shell doesn't have, so it would fail on every run. Fable only.
A third opinion, after the fact
On 21 October one repository moved its second reviewer to a Sonnet model. In the four days after that, 36 changes merged with Sonnet and Opus both approving. On 28 October I had Fable review each of those exact commits, once and blind. It got the same prompt and tools as a live review, saw no later history, and posted nothing.
On 23 October nine test files had to be deleted from the main branch because they failed on every run and kept the build red. All nine came in through these 36 changes, and both reviewers had approved every one.
Fable would have blocked five of the nine. It withheld approval on only five of the 36 changes, and those five are exactly them. On the other four it named the cause of the failure but rated it below HIGH, so those merges would have gone ahead.
- Of Fable's 126 findings, 75 matched something Sonnet or Opus had already said.
- 4 were HIGHs neither of them raised. All 4 were real when checked against the code.
- Of the 8 MEDIUMs only Fable raised, 4 were real, 3 were partly real and 1 was wrong.
- None of the defects it found that are still on the main branch is HIGH.
- The replay cost about $80 at list price, a little over $2 a commit, and about a fifth of a week's Fable quota.
One caveat: the deletion proves the nine files failed. It doesn't prove each one failed for the reason Fable gave, and I checked that reason in the code for only some of them.
What I take from it
- Neither model covers the other. Each found real defects the other did not mention at all. Running both is what catches them.
- Opus is the better value per review. At least as many unique catches as Fable, at about a quarter of the cost and under half the time. The replay is the case for keeping Fable in one seat. With Fable out of the pair, nine broken changes got through that it would have blocked or flagged.
- Severity is where they differ most. Counting one-model HIGHs, Opus leads 59 commits to 21. Counting real defects the other model never mentioned, it leads 10 to 7.
- The gate matters as much as the reviewers. Every merge with a HIGH still standing came from the process, not from a model missing something.
Method and limits
Data. Every review posted on the lab's forge in the period, with severities taken from each review's machine-readable findings block and the model from its provenance line. A finding withdrawn when the same reviewer re-reviewed the same commit stops counting.
Judging. The matching was done by a Sonnet model, which was neither reviewer, reading the review text. Only the findings it judged unique, defect-shaped and deserving HIGH were checked against the code, so 10 and 7 are lower bounds.
Not measured, mostly. Defects that neither reviewer caught. Replaying a third model, as above, finds only what that model would have caught. Finding the rest needs incidents traced back to the change that caused them, and the lab doesn't record that yet.
A late change. On 21 October one repository moved its second code reviewer to a Sonnet model, the same family that writes the code. Its 58 reviews are in the totals above but not in the two-model tables. The replay above is the only measure of that pairing so far.
The review loop itself: Nine rounds with a machine reviewer. The machinery around it: Plan dispatch, drawn.