We Scored Every Comment Six AI Reviewers Posted

The old scoring credited one bug per pull request

Greptile published a benchmark of 50 real-world pull requests drawn from Sentry, Cal.com, Grafana, Keycloak and Discourse. Each case nominates one bug, and each tool is scored on whether it found that bug. Greptile came top at 82%, ahead of BugBot at 58%, GitHub Copilot at 54% and CodeRabbit at 44%.

Two things are wrong with that, and neither is about who ran it. The first is that scoring one nominated bug rewards volume: post three times as many comments and you get three times as many chances at it, while nothing is ever charged for what else you said. The second is that the nominated bug is not the only bug. When we had every comment from every tool adjudicated, the same cases turned out to contain 183 distinct real defects. Scoring the one and ignoring the rest is close to a coin toss on which tool looks best.

The scoreboard

Every comment from five competitors and our own reviewer, shown with the diff to two independent judge models, classified independently, and clustered so that a defect several tools reported counts once. A comment becomes evidence only where both judges agree. Across 51 cases that yields 183 distinct real defects, and recall is measured against those.

Tool	Defect recall	Case recall	Precision	Comments	Nitpicks	Wrong
CodePulse Reviewsours	76.5%	75.5%	89.4%	345	18	15
Greptile	37.2%	79.6%	36.5%	316	111	56
BugBot (Cursor)	28.4%	71.4%	81.2%	81	3	10
CodeRabbit	23.5%	49.0%	44.8%	143	46	12
GitHub Copilot	18.6%	51.0%	35.1%	179	53	45
Graphite	4.4%	12.2%	84.6%	31	3	1

Defect recall is the share of the 183 known defects a tool found. Case recall is the share of the 49 cases where it landed at least one real finding. Precision counts nitpicks against a tool, and excludes the comments the two judges disagreed about.

Read the defect column first. CodePulse found 140 of the 183 real defects, 76.5%. Greptile, top of the published benchmark, found 68, 37.2% - not far off half as many. Every other tool trails further behind. Then read the two noise columns next to it.

Precision is where the gap is a different category

89.4% of our comments described a real defect. 36.5% of Greptile's did. That is not a few points, it is a different kind of tool. Of the 316 comments Greptile posted, 96 described a real defect, 111 were nitpicks and 56 were simply wrong. Of the 345 we posted, 277 described a real defect, 18 were nitpicks and 15 were wrong. Per pull request that is 3.4 pieces of noise from Greptile against 0.7 from us.

Finding more than twice the bugs while being wrong a quarter as often is not a close call. The noise is the part your team pays for, and it is the part nobody publishes.

The rest of the field sorts the same way. BugBot is the other disciplined reviewer: 81.2% precision on 81 comments, and 28.4% defect recall. CodeRabbit and Copilot both post more than a hundred comments to find fewer defects than either of us, at 44.8% and 35.1% precision. Graphite is accurate when it speaks and almost never speaks: 84.6% precision, 4.4% defect recall.

Why we do not file nitpicks

A reviewer can always say something. Unused imports, a name that could be clearer, a docstring that could be fuller, a spelling mistake in a comment - a language model will generate those endlessly and every one of them will be technically correct. That is exactly why they are worthless as a signal of a tool working, and exactly why a benchmark that only asks "did it find the bug" will reward a tool for emitting them.

The cost is not the seconds spent reading one. It is what a steady drip of correct, unimportant comments does to how the next one gets read. Once a reviewer has trained your team to resolve without reading, the critical finding at comment 14 gets closed with the other thirteen. We would rather be quiet and be read.

The number we do not lead on

Greptile edges us on case recall, 79.6% to 75.5%. That means it lands at least one real finding in a couple more of the 49 cases than we do, and that is a genuinely useful property: a reviewer that says something true about every pull request never leaves anyone wondering whether it was awake. Volume buys that, and it is the one thing volume buys. It costs 111 nitpicks and 56 wrong comments to get there, and we would rather close that couple of cases without buying the noise with them.

The fixture leaked. We sealed ours.

The public evaluation forks give away their own answers in three ways, and we found all three by auditing our own runs:

competitor reviews deflates a score
The evaluation forks still carry rival bots’ reviews, and the agent could read them with an ordinary GitHub CLI call. On one case ours found the planted defect, saw it had already been reported, and suppressed its own comment as a duplicate - scoring a miss on a bug it had found. The GitHub CLI is withheld from the agent by a PATH shim. The harness fetches CI status itself, so nothing legitimate needs the binary.
branch names inflates a score
The fork branches are named after the defect they plant, for example one called error-upsampling-race-condition, and the review prompt rendered the branch names. Head and base refs are replaced with neutral placeholders before the prompt is built.
repository history inflates a score
The fixture shared the clone’s object store, so ordinary git commands could reach the other branch names, later commits, and the upstream commit that repairs the bug. The fixture is now built from an empty repository with an ancestry-only, depth-limited fetch, so the fix commit is genuinely absent. A test asserts it cannot be resolved.

Our figures above come from runs with all three sealed. Note which way they run: the first worked against us, the other two inflate whoever is exposed to them - and the competitors' comments were harvested from the unsealed public forks.

That is the asymmetry worth naming, and we want to be precise rather than insinuating. The competitors' comments in this study were harvested from those public forks, where the branch is named after the planted bug and the repository history contains the upstream commit that fixes it. We have no idea what any of those tools read, and no way to find out. We do know what ours read, because we removed the channels and then checked the transcript. We are not alleging anyone cheated. We are saying that only one of the six columns in this study was produced under conditions anybody verified, and it is ours.

What this does not establish

About 16% of comments are parked as disputed - the two judges could not agree on them - and are excluded from precision entirely rather than being counted for or against.
The golden list can only contain defects some reviewer commented on. A bug that all six tools missed is invisible to it. This is a fair basis for comparing tools and it is not an estimate of how many bugs are in the code.
Two channels are still open. These are public repositories, so plain HTTPS to the GitHub API still serves those rival reviews to anything that reaches out for them. The clones and the answer key are readable on the host running the benchmark. Closing them means running the benchmark inside the production worker image with egress control, which is the next piece of work.
We built the harness, chose the metrics and ran our own arm. Every competitor figure here comes from comments they posted publicly; every figure of ours comes from a harness we control. That is exactly why the method, the caveats and the per-comment verdicts are published rather than summarised.

Why any of this matters to a team choosing one

Because the noise is the part you will feel and the part nobody publishes. CodePulse stores the full body of every review, bot reviews included, so you can measure your own reviewer rather than trusting anyone's table, ours included: what share of its comments get addressed, whether review turnaround moved after you adopted it, and whether the defect rate followed.

For pricing, capability and the rest of the field, see the AI code review bots comparison.