The old scoring credited one bug per pull request
Greptile published a benchmark of 50 real-world pull requests drawn from Sentry, Cal.com, Grafana, Keycloak and Discourse. Each case nominates one bug, and each tool is scored on whether it found that bug. Greptile came top at 82%, ahead of BugBot at 58%, GitHub Copilot at 54% and CodeRabbit at 44%.
Two things are wrong with that, and neither is about who ran it. The first is that scoring one nominated bug rewards volume: post three times as many comments and you get three times as many chances at it, while nothing is ever charged for what else you said. The second is that the nominated bug is not the only bug. When we had every comment from every tool adjudicated, the same cases turned out to contain 160 distinct real defects. Scoring the one and ignoring the rest is close to a coin toss on which tool looks best.
The scoreboard
Every comment from five competitors and our own reviewer, shown with the diff to two independent judge models, classified independently, and clustered so that a defect several tools reported counts once. A comment becomes evidence only where both judges agree. Across 39 cases that yields 160 distinct real defects, and recall is measured against those.
| Tool | Defect recall | Case recall | Precision | Comments | Nitpicks | Wrong |
|---|---|---|---|---|---|---|
| CodePulse Reviewsours | 76.9% | 94.6% | 94.0% | 170 | 8 | 0 |
| Greptile | 34.0% | 78.4% | 33.5% | 226 | 80 | 47 |
| BugBot (Cursor) | 28.2% | 70.3% | 84.2% | 65 | 2 | 7 |
| CodeRabbit | 23.7% | 51.4% | 45.3% | 113 | 36 | 11 |
| GitHub Copilot | 18.6% | 48.6% | 35.3% | 144 | 49 | 37 |
| Graphite | 4.5% | 13.5% | 80.8% | 31 | 3 | 2 |
Read the defect column first. CodePulse found 120 of the 160 real defects, 76.9%. Greptile, top of the published benchmark, found 53, 34.0% - not far off half as many. Every other tool trails further behind. Then read the two noise columns next to it.
- CodePulse Reviews(ours)170 comments
- Greptile226 comments
- BugBot (Cursor)65 comments
- CodeRabbit113 comments
- GitHub Copilot144 comments
- Graphite31 comments
Precision is where the gap is a different category
94.0% of our comments described a real defect. 33.5% of Greptile's did. That is not a few points, it is a different kind of tool. Of the 226 comments Greptile posted, 64 described a real defect, 80 were nitpicks and 47 were simply wrong. Of the 170 we posted, 125 described a real defect, 8 were nitpicks and 0 were wrong. Per pull request that is 3.3 pieces of noise from Greptile against 0.2 from us.
Finding more than twice the bugs, without a single comment the judges called wrong, is not a close call. The noise is the part your team pays for, and it is the part nobody publishes.
The rest of the field sorts the same way. BugBot is the other disciplined reviewer: 84.2% precision on 65 comments, and 28.2% defect recall. CodeRabbit and Copilot both post more than a hundred comments to find fewer defects than either of us, at 45.3% and 35.3% precision. Graphite is accurate when it speaks and almost never speaks: 80.8% precision, 4.5% defect recall.
Why we do not file nitpicks
A reviewer can always say something. Unused imports, a name that could be clearer, a docstring that could be fuller, a spelling mistake in a comment - a language model will generate those endlessly and every one of them will be technically correct. That is exactly why they are worthless as a signal of a tool working, and exactly why a benchmark that only asks "did it find the bug" will reward a tool for emitting them.
The cost is not the seconds spent reading one. It is what a steady drip of correct, unimportant comments does to how the next one gets read. Once a reviewer has trained your team to resolve without reading, the critical finding at comment 14 gets closed with the other thirteen. We would rather be quiet and be read.
The number we used to lose
Every previous version of this page had a section here conceding case recall to Greptile: it landed a real finding in a couple more pull requests than we did, and volume bought it that. On this run it has reversed, and by a wide margin - 94.6% of the 39 cases against Greptile’s 78.4%. We now say something real about all but two of them.
We are stating that as a reversal rather than quietly dropping the old paragraph, because anyone who read this page a fortnight ago was told the opposite. The rest of that argument still stands: it costs Greptile 80 nitpicks and 47 wrong comments to get where it gets, and a reviewer that says something about every pull request is only useful if the something is true.
Is the headline in-sample?
Partly, and here is the number that settles it. Our shipping reviewer carries a checklist written from misses on this same corpus, so we scored the 27 of these 39 cases the checklist was demonstrably not written from, separately. On those it finds 80.0% of the defects - higher than its 76.9% across all of them - and lands a real finding in every single one. The tuning is not what is carrying the result, which is the only reason we are comfortable leading with the headline.
Two limits on the other side. This run covers 39 of the 51 cases in our corpus, and on the 12 it does not cover our margin over the field has historically been narrower, so read the gap as measured on this subset. And it is one run per arm with no repeats, in a benchmark we know to be noisy run to run. Every gap in the table above is many times wider than that noise; a difference of a few points would not be.
The fixture leaked. We sealed ours.
The public evaluation forks give away their own answers in three ways, and we found all three by auditing our own runs:
- competitor reviews deflates a score
The evaluation forks still carry rival bots’ reviews, and the agent could read them with an ordinary GitHub CLI call. On one case ours found the planted defect, saw it had already been reported, and suppressed its own comment as a duplicate - scoring a miss on a bug it had found. The GitHub CLI is withheld from the agent by a PATH shim. The harness fetches CI status itself, so nothing legitimate needs the binary. - branch names inflates a score
The fork branches are named after the defect they plant, for example one called error-upsampling-race-condition, and the review prompt rendered the branch names. Head and base refs are replaced with neutral placeholders before the prompt is built. - repository history inflates a score
The fixture shared the clone’s object store, so ordinary git commands could reach the other branch names, later commits, and the upstream commit that repairs the bug. The fixture is now built from an empty repository with an ancestry-only, depth-limited fetch, so the fix commit is genuinely absent. A test asserts it cannot be resolved.
Our figures above come from runs with all three sealed. Note which way they run: the first worked against us, the other two inflate whoever is exposed to them - and the competitors' comments were harvested from the unsealed public forks.
That is the asymmetry worth naming, and we want to be precise rather than insinuating. The competitors' comments in this study were harvested from those public forks, where the branch is named after the planted bug and the repository history contains the upstream commit that fixes it. We have no idea what any of those tools read, and no way to find out. We do know what ours read, because we removed the channels and then checked the transcript. We are not alleging anyone cheated. We are saying that only one of the six columns in this study was produced under conditions anybody verified, and it is ours.
What this does not establish
- About 16% of comments are parked as disputed - the two judges could not agree on them - and are excluded from precision entirely rather than being counted for or against.
- The golden list can only contain defects some reviewer commented on. A bug that all six tools missed is invisible to it. This is a fair basis for comparing tools and it is not an estimate of how many bugs are in the code.
- Two channels are still open. These are public repositories, so plain HTTPS to the GitHub API still serves those rival reviews to anything that reaches out for them. The clones and the answer key are readable on the host running the benchmark. Closing them means running the benchmark inside the production worker image with egress control, which is the next piece of work.
- We built the harness, chose the metrics and ran our own arm. Every competitor figure here comes from comments they posted publicly; every figure of ours comes from a harness we control. That is exactly why the method, the caveats and the per-comment verdicts are published rather than summarised.
Why any of this matters to a team choosing one
Because the noise is the part you will feel and the part nobody publishes. CodePulse stores the full body of every review, bot reviews included, so you can measure your own reviewer rather than trusting anyone's table, ours included: what share of its comments get addressed, whether review turnaround moved after you adopted it, and whether the defect rate followed.
For pricing, capability and the rest of the field, see the AI code review bots comparison.
Measure your own review noise
Track how many bot comments your team actually addresses, what happened to review turnaround after you adopted a reviewer, and whether the defect rate followed. Five-minute setup, no per-seat pricing.
Start free