What 6,000+ review runs taught us about useful feedback

Measure useful findings.

Every false positive asks a developer to spend time disproving a claim. Counting comments alone rewards volume. We care about findings worth acting on: concrete issues supported by the code, with a clear reason to fix them.

We used real PRs from our work on snaptale.com to test an internal version of HeyDeer in everyday development. We reviewed our internal PR activity across projects over the past six months, from March 19 to September 19, 2026. This window contains 6,132 recorded review attempts, including 5,879 successful runs. This includes retries, failures, cancellations, and different configurations; it is not a count of unique PRs or independently scored test cases.

The Elo analysis uses a fixed set of 1,024 assessed internal PR review rounds from September 5–14. Of these, 752 rounds produced eligible comparisons.

The model is one part of the review.

A model generates the analysis; its harness provides tools, repository access, prompts, and execution control. We identify both. In these runs, Grok 4.6 used Grok Build, Opus 5 used Claude Code, and GPT-5.6 Sol used Codex.

Each baseline received one review assignment. “Single pass” here does not mean one model response: the harness can make tool calls and perform multiple internal steps. HeyDeer’s deeper review workflow adds independent reviewers, then a lead reviewer verifies and deduplicates their findings before publication. These experiments with an internal version of HeyDeer helped shape that approach.

Observed reviewer ratings

These are relative Elo ratings for selected configurations within the archived comparison pool. Ratings use the full comparison pool, available in the downloadable results.

One round may generate several pairwise comparisons. Sample sizes differ, and Sol has no direct matchup with Grok in this archive. Shared opponents connect the ratings. These numbers do not establish a universal model ranking.

The HeyDeer row describes the multi-round architecture. Its system-level Elo is unmeasured: this archive scores individual reviewers and contains no independently judged entry for the completed workflow.

Reading the ratings table

The table shows four of the six configurations in the comparison pool. GPT-5.6 Sol finished highest at 1,393, followed by Opus 5 at 1,244, Grok 4.6 at 1,106 and Sonnet 5 at 919. The other two configurations stay in the downloadable results, and they still shape these numbers, because every rating comes from the same replay.

Sample sizes vary a lot. Opus 5 had eligible matches in 625 rounds (702 pairwise comparisons), Grok 4.6 in 206 rounds (320), GPT-5.6 Sol in 86 rounds (86) and Sonnet 5 in 62 rounds (114). A rating built on 86 rounds moves more with each new result than one built on 625, so treat the top of the table as its least settled part.

Counting a tie as half a win, here’s the share of available points each configuration took: Sol 64.5% (49 wins, 13 ties, 24 losses), Opus 5 51.3% (343–34–325), Grok 4.6 38.3% (110–25–185) and Sonnet 5 28.5% (28–9–77). The order matches the table, but each share mixes different opponents.

Who played whom matters too. All 86 of Sol’s comparisons were against Opus 5. Opus 5 won 59 of its 79 direct comparisons with Grok 4.6 and tied 3. Grok 4.6 and Sonnet 5 met 58 times: Grok won 32, tied 5 and lost 21. Opus 5 and Sonnet 5 never met directly, so the 324-point gap between them rests entirely on shared opponents.

The Elo formula turns a rating gap into an expected score. Sol’s 150-point lead over Opus 5 implies about 70% of the points; Sol took 64.5% in their direct matches. Opus 5’s 138-point lead over Grok 4.6 implies about 69%, and Opus took 76.6%. Grok’s 187-point lead over Sonnet 5 implies about 75%, against 59.5% observed. Final ratings reflect every match in chronological order, so they won’t line up exactly with any single head-to-head record.

From 6,132 runs to 1,001 comparisons

Of the 6,132 recorded attempts, 5,879 succeeded (about 96%), 189 were canceled, 63 failed and one was still running when we queried the records. Those attempts include retries and different configurations, so they describe activity, not test cases.

The Elo sample is much narrower: 1,024 assessed rounds from September 5–14. 272 of them (about 27%) produced no eligible pair. A round drops out when the review wasn’t published, the code changed between dispatch and the final revision, fewer than two reviewer configurations succeeded, or no pair had an effective finding or a false positive. The remaining 752 rounds produced 1,001 pairwise comparisons, about 1.33 per rated round. Each comparison has two sides, so the per-configuration counts across the full pool add up to 2,002.

How a finding becomes a score

• Effective findings: issues judged valid and either published or fixed during the review. Repeated, low-value, and unresolved findings do not earn this credit.
• Exclusive findings: effective issues found by only one reviewer, with no independent discovery by the lead reviewer. Exclusivity is checked against the full round’s recorded contributors.
• False positives: findings explicitly judged incorrect. Missing feedback or an unpublished comment is not automatically a false positive.
• LLM feedback: the lead reviewer supplies structured judgments and reasons. This is internal model-assisted adjudication, not independent human ground truth.

The higher score wins the pair; a tie counts as half a win. Both reviewers must succeed on the same dispatched and final code revision, and the review must be published. Pairs with no effective findings and no false positives are skipped.

Every reviewer starts at 1,200. We replay rounds chronologically with K = 32 and a 400-point scale. All matches within a round use the ratings from before that round, then their updates are applied together.

A worked example of one round

The public file contains aggregates, not individual rounds, so this example uses made-up findings with the real formula. Reviewers A and B both succeed on the same revision of a published review. A reports three findings the lead reviewer judges valid and publishes, one of which no other reviewer in the round found, plus one finding judged incorrect. B reports two valid, published findings and nothing incorrect.

A wins the pair. If both start at 1,200, each expected score is 0.5, so A gains 32 × (1 − 0.5) = 16 points and B loses 16. Now put A at 1,393 and B at 1,244, close to the Sol and Opus 5 ratings. A’s expected score becomes 1 / (1 + 10^((1,244 − 1,393) / 400)), or about 0.70. A win adds only 32 × 0.30 ≈ 9.5 points, a tie costs about 6.5, and a loss costs about 22.5. A leader’s rating climbs slowly and drops quickly after an upset.

The weights also shape outcomes. One valid finding earns ten points and one false positive costs one, so false positives change the result only when two reviewers’ finding credit is tied or within a few points. When a round has three or more eligible reviewers, each pair counts as its own comparison, and a reviewer that beats two opponents collects both gains together after the round.

Why we use multiple review stages

Independent reviewers can surface different issues. Verification then checks the evidence, reconciles overlapping findings, and filters unsupported claims before they reach the developer. Our implementation gives deeper reviews separate investigation and verification stages for this reason.

The archive measures individual reviewer contributions. Quantifying the improvement attributable to multiple stages requires a matched single-pass versus multi-round experiment, with independent judging and comparable budgets. That comparison remains outside the scope of these results.

Quality and cost need separate evidence.

Elo measures relative outcomes under this rubric; it does not include price, latency, or token use. A higher Elo therefore does not prove better cost-effectiveness. Our product lets teams choose review depth to suit the change. A cost comparison needs matched workloads and measured cost per useful finding.

These retrospective runs differ in repository, date, context, reasoning settings, and reviewer availability. Judgments may inherit the lead reviewer’s biases, and the archive does not label every possible bug. We do not infer recall, a universal accuracy rate, or statistical significance from this table.

What teams can take from this

Score findings, not comments. A rubric that separates valid, published findings from incorrect ones gives you a number you can compare across tools. Our weights of 10, 5 and −1 are one choice; a team with less patience for noise could weight false positives more heavily.

Compare on the same code. Our pairs count only when both reviewers ran on the same dispatched and final revision. Without that rule, one reviewer could be judged against code that had already changed.

Check sample sizes and matchups. Before trusting a gap, look at how many rounds each reviewer has and who it actually played. In our archive, the top-rated configuration played a single opponent over 86 rounds, while Opus 5 had 625.

Measure cost on its own. Track price, latency and developer time per useful finding alongside quality. Our guide to evaluating AI code review turns these checks into a trial plan.