필사 모드: Is AI Code Review Actually Worth It — What Measured Evidence Says About Accuracy and False Positives
English- Introduction — Why "AI Found N% of the Bugs" Means Nothing on Its Own
- How Often Are AI Review Comments Actually Acted On — 0.9% or 73.8%?
- Why the Same Metric Spans 80x
- How Expensive Are False Positives — Tencent's 76%
- How Google Has Managed This for Almost 20 Years
- Do Review Bots Kill PRs — Noise and PR Abandonment
- What Are the Vendor Numbers Measuring
- Where the Measurement Itself Breaks Down — "Resolved" Is Not Ground Truth
- So How Do Teams Actually Set This Up
- When Not to Use It
- Decision Rules
- Closing
- References
Introduction — Why "AI Found N% of the Bugs" Means Nothing on Its Own
Let me give the conclusion first. As of July 2026, there is effectively no publicly available, vendor-independent measurement of the precision and false-positive rate of a general-purpose AI code reviewer in real production. Only three kinds of things have been published — benchmarks the vendors built themselves, proxy metrics that stand in by measuring whether developers reacted to a comment, and a single-company case study. This post takes each of the three apart.
And that fact by itself matters in practice. A claim that "the AI reviewer caught N% of the bugs" can't be judged unless it comes with two things.
- On what dataset it was measured, and who built the ground truth and how
- What the false-positive rate is — that is, how many pieces of nonsense a developer had to read to get that N%
Recall without the second is free to push to 100% — just comment "there might be a bug here" on every line. A review tool's value comes not from what it catches, but from what it catches while not generating noise. So below, every time a number appears, I check the denominator and the source of the ground truth first.
This post carries the concern of Separating Signal from Noise When You Evaluate AI Coding Models over to code review. There the subject was benchmarks; here it's review comments — but the principle, "ask first what the number is measuring," is the same.
How Often Are AI Review Comments Actually Acted On — 0.9% or 73.8%?
The most widely used proxy metric is "did the developer change the code the way the comment said?" There are two public measurements using this metric, and they collide head-on.
Measurement A — open source (Gan et al., arXiv 2508.18771). They picked 16 AI code review actions for GitHub workflows and collected 22,326 AI-generated review comments from 178 mature repositories (each with 50+ PRs). GPT-4.1 judged 5,652 of them for validity, classifying 3,955 as "valid (specific and actionable)," after which o3-mini judged whether each was actually acted on. The results:
- Share of valid AI comments that led to an actual code change: 0.9~19.2% depending on the tool
- Human review comments by the same criterion: 60%
- Hunk-level actions: 6.5~19.2% / file-level actions: 0.9~4.2%
- Highest was
coderabbitai/ai-pr-reviewerat 19.2%, lowestmattzcarey/code-review-gptat 0.9%
Measurement B — inside one company (Cihan et al., arXiv 2412.18531). This is the case of Beko, a multinational home-appliance maker, adopting a review bot (GPT-4 Turbo) based on the open-source Qodo PR-Agent. 238 people across 10 projects had access, and 4,335 PRs from 3 of those projects (1,568 of which got an automated review) were analyzed. After filtering, the tally is of 1,408 comments on merged PRs out of 4,408 bot comments.
- Comments labeled "Resolved": 73.8% ("Won't Fix" was 21.3%)
- Ranged from 55% to 90% across projects
A metric of the same name sits somewhere between 0.9% and 73.8%. That's up to an 80x difference. Neither one is wrong — they're just measuring different things.
Why the Same Metric Spans 80x
The difference comes from measurement conditions, not tool performance. Four things differ.
First, the denominator differs. Measurement B's 73.8% counts only comments on merged PRs. Comments on abandoned PRs are excluded from the tally to begin with. Because cases where a bad review killed the PR never enter the denominator, this number is structurally biased upward. Measurement A, by contrast, looks at valid comments across entire mature open-source repositories.
Second, the force that produces the label differs. At Beko, to merge a PR a developer must explicitly finalize the status of every review comment. So "Resolved" is not only voluntary agreement but also a procedural act to pass the merge gate. Open-source contributors face no such compulsion.
Third, the party that makes the ground truth differs. Measurement A's "acted on" label was inferred by an LLM (GPT-4.1 + o3-mini) reading the commits, not by a human (they report validating against a hand-annotated subset). Measurement B's labels were applied by the developers themselves.
Fourth, the tools and models differ. Measurement A spans 16 GitHub Actions; Measurement B is a single Qodo PR-Agent.
The practical rule here is simple. When you see a "N% comment-adoption rate," ask first whether it counted only merged PRs, whether the label was compelled, and who built the ground truth. Without those three, the number can't be compared to another organization's number. It can't even be compared to your own team's.
How Expensive Are False Positives — Tencent's 76%
The claim that false positives are "most of the cost" comes up often, but actual measured numbers are rare. One came from Tencent (arXiv 2601.18844, January 2026).
Tencent runs its in-house static analyzer BkCheck on the codebase of its Advertising and Marketing Services (AMS) division. From real alarms between September 2024 and June 2025, they selected those with sufficient code context and built a dataset of 433. The composition is the key.
- 328 false positives, 105 real bugs → a false-positive rate of 76%
- The paper adds a caveat here. Once you account for cases excluded for incomplete code context, the authors estimate the real false-positive rate exceeds 90% (this is an author estimate, not a measurement)
- Manual review time per alarm: about 10 minutes per round, up to 2 rounds → 10~20 minutes (based on interviews with 5 developers and reviewers)
Why so high? The paper's explanation is a design choice — to avoid missing bugs, enterprise static analyzers prioritize recall over precision. So they conservatively warn even about infeasible paths, and end up with a higher false-positive rate than even open-source analyzers.
Do the arithmetic and the cost structure appears. Out of 100 alarms, 76 are junk, and at 10~20 minutes each that's 12.7~25.3 hours on the junk alone. The value of the 24 caught has to exceed that time for the tool to come out ahead. Not "found N bugs" — this subtraction is the actual profit-and-loss statement.
What Happens When You Use an LLM as a False-Positive Filter
The same paper goes on to measure something interesting. What happens if you ask an LLM "is this alarm real?" This is with only a base prompt.
| Technique | Model | Accuracy | Precision | Recall | F1 |
|---|---|---|---|---|---|
| Base prompt | GPT-4o | 0.50 | 0.28 | 0.66 | 0.39 |
| Base prompt | Claude-Opus-4 | 0.40 | 0.26 | 0.83 | 0.40 |
| Base prompt | Qwen-3-Coder | 0.55 | 0.29 | 0.62 | 0.40 |
| Base prompt | DeepSeek-R1 | 0.43 | 0.27 | 0.84 | 0.41 |
| LLM4PFA (with static analysis) | GPT-4o | 0.93 | 0.93 | 0.75 | 0.83 |
| LLM4PFA (with static analysis) | Claude-Opus-4 | 0.93 | 0.83 | 0.88 | 0.85 |
| LLM4PFA (with static analysis) | Qwen-3-Coder | 0.94 | 0.93 | 0.79 | 0.86 |
| LLM4PFA (with static analysis) | DeepSeek-R1 | 0.93 | 0.84 | 0.86 | 0.85 |
Look at the precision in the base-prompt rows. It sits between 0.26 and 0.29. But in this dataset the base rate of real bugs is 105/433 = 24.2%. That is, a dumb classifier that answers "real" to every alarm has a precision of 24.2%. The four state-of-the-art models come out just 2~5 percentage points above the base rate. That's closer to noise than to signal.
By contrast, LLM4PFA — which combines path constraints extracted from static analysis with LLM reasoning — jumps to 0.93~0.94 accuracy. In the paper's words it removes 94~98% of false positives. What matters is that this result came from changing the structure, not from changing the model.
That said, the abstract's phrase "while maintaining high recall" needs careful reading against the table. LLM4PFA's recall is 0.75~0.88. That means 12~25% of real bugs get caught in the filter and quietly disappear. The authors know this, and they write that since enterprises prioritize recall, a Claude (0.88) or DeepSeek-R1 (0.86) pairing is a better fit in practice. A false-positive filter is not free — it's a trade against false negatives.
The cost is 2.1~109.5 seconds per alarm, or $0.0011~$0.12 in money. Compared with the 10~20 minutes a human spends, that's a different order of magnitude. The limits are clear too — the data is confined to Tencent C/C++ code, and for confidentiality the dataset is not released.
How Google Has Managed This for Almost 20 Years
The most concrete public material on false-positive governance is still from the pre-LLM era: Google's Tricorder paper (Sadowski et al., Communications of the ACM, April 2018, Vol. 61, No. 4). There's not a single line about LLMs, but for this post's subject it is, if anything, more precise.
The core is the definition of an "effective false positive." In the paper's terms — if a developer saw the issue and took no action, it is an effective false positive. If the analyzer made an incorrect point but the developer fixed it anyway for readability, that is not an effective false positive; conversely, if the analyzer pinpointed a genuine defect but the developer didn't understand it and let it pass, that is an effective false positive. The paper states the reason for the distinction explicitly — it is the developer, not the tool's author, who determines the tool's false-positive rate.
On top of that Google built its rules.
- Checks surfaced at the code-review stage are allowed up to 10% effective false positives (compile-time checks are stricter)
- Four conditions for a review check: it must be understandable; actionable and easy to fix; below 10% effective false positives (developers must feel it points at a real problem at least 90% of the time); and have a real impact on code quality
- Tricorder tracks the ratio of reviewers' "Please fix" to "Not useful" clicks. If that ratio exceeds 10%, the analyzer gets turned off — until its author improves it
- A "Not useful" click automatically files an issue and routes it to the team that owns the analyzer
The scale as of January 2018 is public too. Around 50,000 code-review changes were analyzed per day, peaking at 3 analyses per second. Reviewers clicked "Please Fix" more than 5,000 times a day, automated fixes were applied about 3,000 times a day, and "Not useful" clicks numbered 250 a day. 250 divided by 5,000 is 5% — below their own 10% threshold. The numbers line up with the policy.
Two things to take away. One is that a concrete 10% baseline already exists; the other is that a baseline only means something if there's a mechanism to turn things off. Today no AI review vendor says "our tool's effective false-positive rate is X%, and if it goes over we turn the check off automatically."
And to be honest, Google's definition has a weakness we'll see later — an effective false positive is defined by developer behavior, and developer behavior is not always a quality judgment.
Do Review Bots Kill PRs — Noise and PR Abandonment
The most recent measurement is a study in MSR '26 (arXiv 2604.03196, DOI 10.1145/3793302.3793614). From 19,450 PR review comments in the AIDev dataset, they extracted 3,109 PRs reviewed by an actual code review agent (CRA), and looked at the 2,456 in "Commented" status.
| Reviewer setup | PRs | Merge rate | Close (abandon) rate |
|---|---|---|---|
| CRA only | 281 | 45.20% (127) | 34.88% (98) |
| Human only | 1,176 | 68.37% (804) | 21.60% (254) |
The gap is 23.17 percentage points, and the chi-square test gives a statistic approaching 84 (83.0319, df 8) with p < 0.001.
The authors then classified the comments on the 98 abandoned CRA-only PRs by "signal-to-noise ratio." They put runtime errors, crashes, compile failures, API breakage, and security vulnerabilities as primary signal, and architecture, performance, and maintainability issues as secondary signal, then computed the ratio of signal comments to all comments (two researchers classified independently, Cohen's kappa 0.75).
- 59 of the 98 (60.2%) fell in the 0~30% signal-ratio band — more than two-thirds of the comments were not actionable
- 31~59%: 14 (14.3%) / 60~79%: 7 (7.1%) / 80~100%: 18 (18.4%)
- 12 of 13 CRAs (92.31%) had a mean signal ratio below 60%
- Individual tools: Copilot 19.79%,
github-advanced-security[bot]27.62%,codefactor-io[bot]0.00%,entelligence-ai-pr-reviews[bot]52.29% (7),cursor[bot]43.40% (5)
The numbers are striking, but it matters to carry over this study's limitations intact. Starting with what the paper itself put in its "threats" section.
- Correlation is not causation — the authors say so explicitly. This design can't distinguish whether the CRA being attached killed the PR, or whether humans stayed away from an already-doomed PR and only the bot showed up
- Signal classification is keyword-based, so it can miss valid feedback that lacks the keywords
- The data is confined to GitHub repositories containing AI-generated code
And let me add one thing the paper doesn't. The signal ratio was measured only on the 98 "abandoned" PRs. The signal ratio of the 127 merged CRA-only PRs was not measured. Without a comparison group, the data doesn't support the conclusion that "low signal ratio causes abandonment" — the signal ratio of the merged PRs could have been just as low. What this study clearly shows is two things: "CRA-only-reviewed PRs are abandoned more often" and "the comments on abandoned PRs were mostly low-signal." The arrow connecting the two is still a hypothesis.
It's worth checking the industry claim this paper targets. The paper writes that "industry reports claim that turning on a CRA means 80% of PRs need no human comment." The original source is Qodo's 2025 State of AI Code Quality report.
What Are the Vendor Numbers Measuring
Open the Qodo report itself and the structure is this.
- The report's backbone is a survey of 609 developers. "81% of people who use AI review saw quality improve," "69% of teams claiming a 10x speedup said quality got better" — all self-reported perception, not measurement
- The 80% in question is not from the survey. It's vendor product data tagged separately as "Qodo product insight," and the sentence is "when the AI review tool is on, 80% of PRs receive no human comment or review." The denominator, the time window, the repository population, and the definition of "human comment" are all undisclosed
- There's another claim in the same form — "high-severity (9~10) issues were found in 17% of PRs." The severity was scored by Qodo's own model
And one fact you can verify. In the entire report, the words "false positive," "precision," and "recall" appear zero times. A flagship vendor report about quality contains neither precision nor a false-positive rate.
Two things to add in fairness here. One is an overstatement on the MSR paper's side. "No human comment was left" and "can be handled without humans" are not the same thing — a PR approved without comments was still seen by a human. The paper criticizing the vendor claim slightly inflated it in the quoting. Both sides need careful reading.
The other is credit to Qodo. Separately from that report, Qodo runs a benchmark with a published methodology (released February 4, 2026, revised March 12).
The Qodo Benchmark — Having a Methodology Is Not the Same as Being Trustworthy
The method is this. Take real, merged, production-grade open-source PRs and inject defects with an LLM. First extract per-repository best-practice rules and plant compliance violations, then plant 1~3 more functional bugs — logic errors, edge cases, race conditions, resource leaks. Run 8 tools (Qodo + 7 competitors) at default settings over the resulting 100 PRs and 580 issues, and score with an LLM-as-a-judge. The benchmark repositories are actually public on GitHub (forks of cal.com, tauri, redis, Ghost, aspnetcore, and others).
The result: Qodo comes first at F1 60.1%. Vendor-measured.
How should you read this number? First, publishing the methodology and the data alone puts it ahead of most competitors. But the design has structural problems.
- The definition of a false positive breaks under this design. Qodo's definition is "flagging an issue not on the ground-truth list is a false positive." But the ground-truth list is the list of injected defects. If a tool finds a real bug that was in the PR to begin with, it's tallied as a false positive. Qodo knows this and says it manually adds naturally occurring issues to the ground truth during a validation step, but a real bug that Qodo's validator missed still counts against the tool. So "precision" here is not the same thing as the false-positive rate a developer feels
- Injected bugs are not naturally occurring bugs. The injector sets the distribution of defects. A tool that shares the injector's "view of what a defect is" is favored. The one who built that injector, and the one competing, are both Qodo
- Scoring is LLM judging. As the very next section shows, the reliability of LLM judging on code review comments has been measured, and the result is not good
- Only it gets two configurations. Competitor tools run at a single default setting, while Qodo reports two configurations, Precise and Exhaustive
- Qodo's explanation for the low precision is that "precision is a dimension you can tune in post-processing, whereas recall is the fundamental constraint." There's something to it, but it's also exactly the argument the side with weak precision would make
Even so, the most quotable fact from this benchmark is elsewhere. In a benchmark it designed and scored itself, the winner's F1 is 60.1%. That's a number from the same company as the marketing line ("80% of PRs need no human"). And by Qodo's own observation most of the other tools cluster at high precision and extremely low recall — a strategy of flagging only the most obvious thing and staying quiet.
Where the Measurement Itself Breaks Down — "Resolved" Is Not Ground Truth
Almost every measurement so far leans on "did the developer react to the comment?" Google's effective false positive, Gan et al.'s adoption rate, Cihan et al.'s 73.8% — all of them. So is that label trustworthy?
An April 2026 paper tested exactly this (arXiv 2604.24525). The subject is that very same Beko data from earlier. On 2,604 bot comments the developers had labeled fixed/wontFix (1,733 fixed, about 66%; 871 wontFix, about 34%), it ran G-Eval and LLM-as-a-Judge in binary and Likert modes each, measuring how well they agreed with the developer labels.
| Model | G-Eval binary | G-Eval Likert | Judge binary | Judge Likert |
|---|---|---|---|---|
| GPT-5.2 | 0.45 | 0.59 | 0.44 | 0.58 |
| GPT-4.1-mini | 0.48 | 0.62 | 0.57 | 0.61 |
| Gemini-2.5-pro | 0.51 | 0.57 | 0.53 | 0.55 |
Agreement of 0.44~0.62. The best score is 0.62, from G-Eval Likert + GPT-4.1-mini. Interestingly, GPT-5.2 — the newest and strongest model — is not the best.
The detailed metrics of that best configuration are the most important part of this post.
| Metric | GPT-5.2 | GPT-4.1-mini | Gemini-2.5-pro |
|---|---|---|---|
| Agreement | 0.59 | 0.62 | 0.57 |
| F1 | 0.72 | 0.76 | 0.68 |
| Precision | 0.67 | 0.66 | 0.67 |
| Recall | 0.78 | 0.90 | 0.69 |
| MCC | -0.0020 | -0.0590 | 0.0106 |
F1 0.76 looks plausible. 62% agreement doesn't look bad either. But the MCC (Matthews correlation coefficient) is all near zero, and some are negative. MCC is a correlation coefficient corrected for class imbalance, and zero means no correlation with the ground truth at all. In the paper's own words, looking only at agreement and F1 overestimates the judge's reliability.
Do the arithmetic directly from the numbers the paper reports and it gets clearer. Since 66.5% of the ground truth is "fixed," a dumb judge that answers "useful" to every comment scores like this.
Dumb baseline (always "useful"):
Precision = 1733/2604 = 0.665
Recall = 1.0
F1 = 2(0.665)(1.0)/(0.665+1.0) = 0.799
Agreement = 0.665
MCC = 0
Best-scoring LLM judge (G-Eval Likert + GPT-4.1-mini):
Precision = 0.66 Recall = 0.90
F1 = 0.76 Agreement = 0.62 MCC = -0.059
The dumb baseline beats the best LLM judge on F1 (0.799 vs 0.76) and on agreement (0.665 vs 0.62). This is my own calculation from the paper's published figures, not a sentence the paper states. But it's exactly the direction of why the paper bothered to report MCC at all ("because the dataset is imbalanced").
The more fundamental question is why. The paper sees the developer label itself as not being ground truth.
- At Beko, the status of every comment must be finalized before a PR merges. That is, labeling is an obligation
- From an interview with a software-engineering director — the wontFix side in particular tends to get picked very quickly, because the priority is usually getting the change into production
- The evaluation pipeline sees only the PR diff and limited metadata, while the developer weighs runtime behavior, architectural constraints, and safeguards outside the PR holistically. So a comment that is technically valid in isolation may in reality already be handled elsewhere, or be something the team deliberately chose to accept
The paper's conclusion — developer behavior (resolve/ignore) reflects not only comment quality but also contextual constraints, prioritization decisions, and workflow dynamics, and so cannot be treated as objective ground truth.
This loops back to Google's definition. Google's effective false positive is defined as "the developer took no action." The Beko study shows that this definition also measures organizational pressure. It's not that Google's definition is wrong — Google defined it that way on purpose to measure developer perception, and it's exactly the right metric for deciding whether to turn a tool off. It just means you must not compare that number across organizations or read it as "accuracy."
Here the skeleton of the whole post shows through. There is still no cheap, automatic way to measure the usefulness of review comments. Proxy metrics are contaminated by organizational pressure, and LLM judges can't beat the base rate. So for the question "is AI review helping our team," for now there is no trustworthy method other than sampling and reading them yourself.
So How Do Teams Actually Set This Up
Put the public evidence together and the setups that work share common traits. Everything below rests on the measurements cited above.
1. Scope it narrowly. A general "review this PR" is the worst. That's the story of the base-prompt LLM at Tencent, whose precision was at base-rate level (0.26~0.29 vs 0.242). By contrast, LLM4PFA — which narrowed the scope of judgment by combining static-analysis path constraints — rose to 0.93~0.94 accuracy. The result of changing the structure, not the model.
2. Attach comments to code locations. In Gan et al.'s measurement, the adoption rate of hunk-level actions (6.5~19.2%) was consistently higher than file-level actions (0.9~4.2%). In the same study, comments that were concise and included a code snippet were adopted more often.
3. Manual triggers beat automatic ones. Also from Gan et al. Comment adoption is higher when a human requested the review. Feedback pushed in when it's unwanted doesn't get read.
4. Set a false-positive budget, and turn it off when you exceed it. Google's 10% is the only validated baseline. What matters is not the number but the mechanism to turn things off — a "Not useful" button, automatic routing to the owning team, and deactivation when the threshold is crossed. Without these three, 10% is just a number on a slide.
5. Sample and read them yourself. Exactly as the Beko follow-up recommends — treat automatic scores not as ground truth but as a weak signal, use them only for triage to prioritize what to inspect, and never for approve/reject decisions. And re-evaluate every time you change the model or the prompt (same paper: agreement wobbles from 0.44 to 0.62 depending on model and rubric).
6. Don't remove the human. The MSR study's practical recommendation is that a CRA should augment, not replace human reviewers. In the same data, reviews with humans mixed in (61.09~67.99%) were far closer to human-only (68.37%) than to CRA-only (45.20%).
When Not to Use It
Honestly, here are the cases where you shouldn't use it.
- If the goal is to eliminate human review. There is no public evidence supporting this. The evidence there is points the other way.
- If the goal is speed. In the Beko case, PR close time rose from 5h 52m to 8h 20m (
p < 0.001). That it split by project matters — project #1 rose from 2h 48m to 4h 38m and #3 from 20h 22m to 30h 51m, but #2 fell from 6h 6m to 3h 7m. On top of that, the number of human review comments per PR did not drop significantly. The paper's conclusion is that it found no consistent evidence of time or effort savings. - If your review culture is already dense. This is the same shape of story as the Rust standard library verification campaign. In a codebase where existing tests and reviews already work well, there's little left for a new tool to pick up. The value shows up where the existing defenses are thin.
- If you use it as a filter in a domain where false negatives are fatal. Remember LLM4PFA's recall of 0.75~0.88. A filter that reduces false positives throws away 12~25% of the real bugs along with them.
- If there's no way to turn it off. A check that no one can turn off even when it's noisy gets ignored by everyone within a few weeks, and from then on the real findings get buried with it. In the Google paper's phrase, "warning blindness."
Decision Rules
I won't end on "it depends." Do this.
- Start with one narrow check. A security rule or a style rule — something whose correctness is mechanically verifiable. Start with a general reviewer and you'll lose the team's trust to the first two weeks of noise, and that's the end of it.
- Before turning it on, set a false-positive budget in writing. Use Google's 10% as the default. If you use a different number, write down why.
- Build the turn-off mechanism the same day. A button equivalent to "Not useful," an owner, and a rule to deactivate when the threshold is crossed. Without this, steps 1 and 2 are meaningless.
- Don't judge by automatic scores. Over four weeks, sample 50~100 comments at random, read them yourself, and rate each useful/useless. That sample is your only trustworthy data. If you want to stand up an LLM-judge pipeline, first check whether the judge beats the majority-class baseline. On the Beko data it didn't.
- Justify it by defects, not speed. Time savings have no evidence. Take the number of defects caught times the cost had each defect reached production, and subtract the time developers spent reading noise. This has to be positive for there to be a reason to keep it on.
- Widening the scope comes only after step 5 comes out positive.
In one sentence: AI code review fails when you use it as a "reviewer," and earns its keep when you use it as a "narrow, turn-off-able check."
Closing
The question at the start of this post was "is AI code review worth it?" Gather all the public measurements together and the answer is not "yes/no" but this.
The evidence is thin. But it's thin in a specific way. There's a fair amount of research, and the direction mostly agrees — in open source, AI comment adoption (0.9~19.2%) falls well short of humans (60%), 76% of static-analysis alarms are junk, and PRs with only a review bot attached are abandoned more often. What's missing is a vendor-independent measurement of a general-purpose AI reviewer's production precision and false-positive rate. In its place stand a vendor's F1 of 60.1% where it won its own benchmark, proxy metrics contaminated by organizational pressure, and LLM judges that can't beat the base rate.
So the next time you see a slide that says "the AI reviewer caught N% of the bugs," ask just two things. What dataset? And what's the false-positive rate? If either has no answer, that N% is saying nothing at all.
And this question isn't only for vendors. You have to ask it of the number that shows up on your own team's dashboard four weeks from now, too.
References
- Does AI Code Review Lead to Code Changes? A Case Study of GitHub Actions (Gan et al., arXiv 2508.18771) — 16 actions, 178 repos, 22,326 comments; adoption 0.9~19.2% vs 60% for humans
- Automated Code Review In Practice (Cihan et al., arXiv 2412.18531) — the Beko case, 73.8% resolved, PR close 5h 52m → 8h 20m
- Understanding the Limits of Automated Evaluation for Code Review Bots in Practice (arXiv 2604.24525) — the same Beko data, 2,604 comments, LLM-judge agreement 0.44~0.62, MCC near 0
- From Industry Claims to Empirical Reality: An Empirical Study of Code Review Agents in Pull Requests (MSR '26, arXiv 2604.03196) — CRA-only merge rate 45.20% vs human-only 68.37%, signal-to-noise analysis
- Reducing False Positives in Static Bug Detection with LLMs: An Empirical Study in Industry (arXiv 2601.18844) — 76% false positives out of Tencent's 433, LLM4PFA removes 94~98% (recall 0.75~0.88)
- Lessons from Building Static Analysis Tools at Google (Sadowski et al., CACM April 2018, Vol. 61 No. 4) — the effective-false-positive definition, the 10% threshold, Tricorder's operating scale
- Qodo — 2025 State of AI Code Quality (vendor report, survey n=609) — the original source of "80% of PRs get no human comment"
- Qodo — How we built a real-world benchmark for AI code review (vendor-measured) — 100 PRs and 580 issues injected, Qodo F1 60.1%
- Separating Signal from Noise When You Evaluate AI Coding Models (related post)
- The Rust Standard Library Verification Campaign Found Zero Memory-Safety Bugs (related post)
현재 단락 (1/157)
Let me give the conclusion first. As of July 2026, **there is effectively no publicly available, ven...