Skip to content
Published on

Does AI Actually Make Developers Faster? What the Measured Numbers Say

🇰🇷KO🇺🇸EN🇯🇵JA🇨🇳ZH
Split View필사 모드
Share
Authors

Introduction — two numbers that disagree

On the question of how much AI coding tools improve productivity, we have two well-designed randomized controlled trials. They reached opposite conclusions.

One says developers with AI were 55.8% faster. The other says they were 19% slower.

And there is a twist. The second number is no longer "current" — the researchers who produced it say so themselves. METR, who measured the 19% slowdown, published a follow-up in February 2026 and mounted a warning banner on their 2025 result: "These results are out of date," and they "no longer reflect the current impact of AI models."

So let me be clear about where this post stands. The 19% is not a wrong number; it is a past number. It was really measured in the first half of 2025, and as a record of that moment it still holds. What is now a misreading is treating it as "using AI today makes you 19% slower." METR hung a banner on their own page specifically to stop you from reading it that way.

The usual move is to pick one of the two. Boosters cite the first number, skeptics cite the second. But both are real experiments producing real results. The interesting question is not "which one is right" — it is "what separated them" and, now, one more: "why is this so hard to measure properly?"

Let me flag the punchline early: the most important finding here is not the 19% slowdown. It never was, and it certainly is not now that METR has put an expiry date on it. The most important finding is that self-perception was wrong — and that finding carries no banner. In an earlier piece on LLM burnout I argued that generation got cheap while verification did not. This post is about what that asymmetry looks like once someone puts a stopwatch on it.

Experiment one — 55.8% faster

In 2023, Peng, Kalliamvakou, Cihon, and Demirer ran a controlled experiment on GitHub Copilot. They recruited 95 professional developers, gave Copilot to half of them, and handed everyone the same task: implement an HTTP server in JavaScript, as fast as you can.

The group with Copilot finished 55.8% faster than the control group. The gains were not evenly distributed either — less experienced developers, older developers, and developers with heavy workloads benefited most. The authors read this as a signal that AI pair programmers could help people transition into software careers.

The number still gets cited constantly. What almost always gets dropped in the citation is what kind of task it was.

  • It is greenfield. There is no existing codebase. No legacy to entangle with, no callers to break.
  • It is self-contained. The whole problem fits in your head at once.
  • The spec is unambiguous. Nobody argues about what an HTTP server is supposed to do.
  • Verification is cheap. Does the server start and respond? You know in seconds.
  • It measured speed, not quality. Who maintains that code in six months is outside the experiment.

So 55.8% is not a lie. It is a number that is true under very specific conditions. The trouble is that most working developers spend most of the day outside those conditions.

Experiment two — 19% slower, in early 2025

In 2025, METR (Becker, Rush, Barnes, Rein) measured exactly that outside. The paper is "Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity." Note the Early-2025 sitting right there in the title. It matters later.

The design: 16 experienced open-source developers brought 246 real issues from repositories they themselves maintain. Each issue was randomly assigned to "AI allowed" or "AI not allowed." Participants mostly used Cursor Pro with Claude 3.5/3.7 Sonnet. The trial ran from February to June 2025.

The decisive variable here is the relationship between the developer and the codebase.

  • The repos are mature — averaging over a decade old, more than a million lines of code, over 22,000 stars.
  • The developers averaged five years working in those repos, with roughly 1,500 commits each.
  • In other words, these people carry an enormous amount of undocumented tacit knowledge about that code.

The result: allowing AI increased completion time by 19%. They got slower. In the authors' words, "allowing AI actually increases completion time by 19% — AI tooling slowed developers down." The confidence interval ran from +2% to +39% and did not include zero, so the direction was statistically clear too.

Every verb in that paragraph is past tense for a reason. This is a measurement of the world as it stood between February and June 2025. And that world has since changed.

What happened next — METR's 2026 follow-up

On 24 February 2026, METR published the results of continuing the same experiment. The post is titled "We are Changing our Developer Productivity Experiment Design." At the same time, the warning banner went up on the 2025 paper. It is not common for a research team to pull down the headline number of its own most-cited work. So let us look at the new numbers straight.

  • Returning developers — speedup of -18%, confidence interval -38% to +9%
  • Newly-recruited developers — speedup of -4%, confidence interval -15% to +9%

Two things have to be read at once here.

First, the point estimates are still negative. Even with early-2026 tooling, the average measured in this setting is still not "faster." So this update is not the story that "it turns out AI does speed developers up after all." Reading it that way just repeats the original misreading in the opposite direction.

Second, and yet, both confidence intervals cross zero. Unlike 2025's +2% to +39%, the 2026 figures contain both slowdown and speedup. Statistically, the sign cannot be pinned down. In 2025 you could assert "they got slower." On the 2026 data you cannot.

But there is something more important than the numbers: the self-selection bias METR reported alongside them.

To run an experiment like this, you have to tell a developer "do this task without AI." By 2026, more of them were refusing. In METR's account, there was a significant increase in developers declining to participate at all, "because they do not wish to work without AI." The same thing happened inside the group that did participate: 30% to 50% of developers told METR they were not submitting some tasks at all, because "they did not want to do them without AI."

Do you see why this is fatal? A task you do not want to do without AI is, almost by definition, a task where AI seems most likely to help. Exactly those tasks get systematically drained out of the sample. METR names the direction of the bias themselves — it "likely biases downwards our estimate of AI-assisted speedup."

So the 2026 figures of -18% and -4% are not, on their own, evidence that "AI slows people down." They are numbers produced by the people who did not refuse to work without AI, on the tasks they themselves judged they could handle without it.

What, then, did METR conclude? They say it is likely that "developers are more sped up from AI tools now" than their early-2025 estimates suggested. And they immediately qualify it — because of the selection effects, "our data is only very weak evidence for the size of this increase."

That is the best answer currently available. It probably got better. We do not know by how much. That is what the team measuring this question more seriously than anyone else in the world concluded after their second attempt. Hold on to that. We come back to it at the end.

The real finding — the perception gap

The 19% is past tense now. But the 2025 study contains another finding that carries no banner — and it is the one this post's advice actually rests on.

  • Before the trial, developers forecast that AI would make them 24% faster.
  • After finishing — after actually living through a 19% slowdown — they estimated they had been 20% faster.
  • For reference, economics experts predicted a 39% speedup and ML experts predicted 38%. Everyone got the sign wrong, not just the magnitude.

This is not a story about bad forecasting. It is a story about bad hindsight. These developers misremembered work they had just personally done, in the opposite direction from the stopwatch. That leaves roughly a 40-percentage-point gap between what was perceived and what was measured.

Why would that happen? The most plausible explanation is simple. Generation is visible; verification is not.

The moment the model emits forty lines in three seconds is vivid and memorable. The twenty minutes you then spend reading those lines, questioning their assumptions, checking them against this repo's conventions, and throwing half of them away is not memorable. It just feels like "working." We grade our day on the dopamine spikes and never write the quiet grind into the ledger.

The METR data supports this reading. Developers accepted fewer than 44% of the code the AI generated. The rest was time spent reading, judging, and discarding — producing nothing. And only about one quarter of participants actually got faster with AI at all.

Let me underline this once more. No expiry banner was hung on this finding. What METR declared out of date was the size of the speedup, not how badly people read their own speedup. The tooling moved on by a generation. There is no evidence anywhere that human self-measurement improved in the meantime.

What separated the two experiments

Compress the difference into one sentence and it reads: AI wins when it supplies context you lack, and loses when it has to catch up to context you already have.

The Copilot participants faced a blank screen. Everything the model filled in was pure gain. The METR participants already carried five years of tacit knowledge in their heads. The model does not have that context, getting it there costs prompting, and whatever comes back must then be checked against that same tacit knowledge. The richer your context, the more expensive it is to bring the model up to speed with you.

The sharpest observation in the METR paper is exactly this: in the 2025 trial, developers slowed down more on the issues they were more familiar with. Familiarity was not a shield. It was a tax.

The 2026 data does not contradict this. Returning developers — the ones who have kept their own repos for years — came in at -18%, while newly-recruited developers came in at -4%. The more familiar group slowed down more. But the two confidence intervals overlap heavily, so calling this evidence would be overreach. Consistent with the hypothesis is as far as it goes, and that is exactly how I will file it.

So the two axes that actually matter in practice are familiarity and cost of verification.

  • Unfamiliar, cheap to verify — the quadrant where AI reliably wins. A language you have never used, an unfamiliar API, one-off scripts, scaffolding. If it is wrong you find out immediately, and the model supplies what you did not know. Use it freely.
  • Familiar, cheap to verify — boilerplate, repetitive patterns. Small gains, small losses. Fine either way.
  • Familiar, expensive to verify — METR's quadrant. Subtle logic, invariants in old code, traps only you know about. Writing it yourself may well be faster.
  • Unfamiliar, expensive to verify — the dangerous quadrant. You cannot review what you do not understand. This is where AI output silently becomes technical debt and the invoice arrives months later. If you are here, the fix is not a better tool. It is learning the domain first.

These four quadrants are still a useful map. But one caveat now has to be stapled to it: this is a hypothesis, not a measured law. Which quadrant you are in is something you have to check for yourself, by measuring.

Zooming out — DORA and Stack Overflow

What does the individual-level illusion look like at organizational scale? The 2025 DORA report (State of AI-assisted Software Development) surveyed roughly 5,000 technology professionals.

Adoption is effectively settled. About 90% of developers use AI, and more than 80% say it has increased their productivity. And DORA found that AI genuinely does increase throughput — so far, so optimistic.

The problem is that the same report finds AI also increases instability. When code gets produced faster and in greater volume, every weak link downstream is exposed. Without control systems — automated testing, mature version control, fast feedback loops — a rise in change volume converts directly into instability.

Hence DORA's central thesis: AI is an amplifier. It magnifies an organization's existing strengths and weaknesses together. Drop it onto a healthy engineering system and performance compounds. Drop it onto a broken one and you ship low-quality work faster. Speed without stability is just accelerated chaos.

The 2025 Stack Overflow Developer Survey (roughly 49,000 respondents) supplies the last piece: usage and trust are moving in opposite directions.

  • 84% are using or planning to use AI tools, up from 76% the previous year.
  • Yet 46% actively distrust the accuracy of AI output — more than the 33% who trust it.
  • Only 3.1% say they "highly trust" it.
  • The top frustration, cited by 66%, is "AI solutions that are almost right, but not quite." Second, at 45%: "debugging AI-generated code is more time-consuming."

"Almost right, but not quite" is the phrase that runs through this entire post. Obviously wrong code is cheap to filter out. The expensive kind is code that is plausible enough to tempt you and wrong enough to cost you — and the labor of filtering it shows up on no dashboard anywhere.

So what should you actually do

None of this means "stop using AI." It means there is real work to do, and it is not the work most people think.

1. Stop trusting the feeling; measure your own work. The lesson of this research is that hindsight lies, so you need records rather than recollection. And METR's 2026 follow-up does not weaken this advice — it strengthens it. If a dedicated research team ran a randomized controlled trial twice and still could not pin down the sign, then your end-of-day impression certainly cannot. The instrument does not need to be fancy. Before you start a task, write down your estimate. When you finish, write down the actual time and whether you used AI. Two weeks of that will tell you more about you than any survey will. An unmeasured productivity gain is just a mood.

2. Choose tasks by familiarity and verification cost. Use the four quadrants above. Be generous with AI where the domain is unfamiliar and checking is cheap; be stingy where the domain is familiar and checking is expensive. And where it is both unfamiliar and expensive to verify, spend on learning before you spend on tooling.

3. Budget review as first-class work. As generation approaches free, the bottleneck migrates entirely to verification. As long as review is treated as chores after the "real work," the added cost stays off the books. I went deeper into this in the LLM burnout piece.

4. Treat vendor numbers the way you treat benchmarks. That is, skeptically. Once you remember which task produced the 55.8%, you will ask the same questions of the next number in a marketing deck: which task, measured by whom, and what got left out. And now there is one more question to ask: when was it measured? On how evaluation numbers get shaky, see separating signal from noise in coding evals.

5. Write for humans. If verification is the bottleneck, then readable code is no longer a matter of taste — it is a throughput problem. Which is exactly the argument in when an AI maintains your code, write for humans anyway.

Honest caveats about these numbers

Since I do not want this read as "the research proves AI is useless," the limitations on both sides deserve to be stated plainly.

Limits of the METR 2025 study. Sixteen participants. That is small. It measured one specific setting — large, mature open-source repositories worked by highly skilled maintainers. 56% of participants had never used Cursor before, and the models are from the first half of 2025, which is more than a generation old by now. The authors themselves explicitly said their result does not show: that AI fails to speed up most developers; that AI fails in domains other than software; that near-future AI will fail in this same setting; or that there is no better way to use today's AI to get a speedup. None of those four are claims of this paper. And the authors have now added an expiry warning on top.

Limits of the METR 2026 follow-up. In one sense these are worse. Self-selection bias is eating the skeleton of the experiment. The more developers refuse to work without AI, the more what remains to be measured skews toward "tasks that are fine to do without AI." It is honest of METR to name the direction of that bias — it pushes the estimate down — but knowing the direction is not knowing the size. Hence the wider confidence intervals and the weaker conclusion. And this problem gets worse over time, not better: the more useful AI becomes, the harder it is to obtain a no-AI control group at all. That is precisely why METR titled the follow-up around changing the experiment design.

Limits of the Copilot study. One task, greenfield, self-contained. It measured completion time — not code quality, not maintainability. It is also worth noting that the author list includes GitHub-affiliated researchers.

Limits of DORA and Stack Overflow. Both are self-report surveys. And for precisely the reason METR uncovered, self-report is the least trustworthy instrument available on exactly this topic.

So the honest conclusion is not "AI slows you down." The honest conclusion is three sentences. The effect is context-dependent enough that even its sign flips. That sign does not come out cleanly even from a well-designed randomized trial. And it certainly does not come out of your intuition.

Closing

We tend to ask whether a tool is good or bad. What this data says is that this is the wrong question. The same tool produced a 55.8% gain in one experiment and a 19% loss in another. The tool did not change. What changed was the kind of work and the context of the person doing it.

And there is another layer on top of that. Even the 19% is now a "then" number rather than a "now" number — flagged as such by the very people who produced it. The follow-up could not pin down the sign, and the reason was not insufficient data. It was that the clean control group is disappearing, because people refuse to work without AI.

You can take that as a reason to be humble, or as a reason to hold the conclusion harder. I think it is the latter. Consider what happened here. A dedicated research team designed a randomized controlled trial, collected hundreds of genuine issues from real repositories, and ran it twice. And what they came away with was: it probably improved, but we have only very weak evidence about the size.

If they could not pin down the sign with a stopwatch in hand, you are not going to get it from a feeling. That is the real conclusion of this post, and the 2026 update did not shake it. It drove the nail deeper.

And the gap — the one where people who were 19% slower believed they were 20% faster — is still sitting exactly where it was. Most of us are still standing exactly where they stood: confident about our own productivity, having never once measured it.

So if you take one thing from this post, let it be this. Use AI more or use it less; it does not much matter. But log your actual working hours for the next two weeks. You have one advantage METR does not: you only have one subject to study, and that subject cannot refuse to participate. That data will be more accurate than any benchmark, any vendor deck, and above all more accurate than how you feel.

References

Discuss on Twitter • View on GitHub