Separating Signal from Noise When You Evaluate AI Coding Models — Why SWE-bench Got Shaky

Introduction — Why the Leaderboard Got Hard to Trust
The Two Axes — Signal and Noise
Where the Noise in Coding Benchmarks Comes From
What Teams Should Actually Do
Closing
References

Introduction — Why the Leaderboard Got Hard to Trust

A coding model's SWE-bench score updates every week, yet the moment you wire it into your own codebase the ranking and the felt experience often diverge. The OpenAI evals team (Frontier Evals) post trending on GeekNews right now, "Separating signal from noise in coding evaluations," takes that gap head on. The conclusion is blunt: SWE-bench Verified, the most widely used coding benchmark, no longer gives meaningful signal, thanks to contamination and design flaws.

This isn't the usual "no benchmark is perfect" shrug. The point is a method — separate signal from noise and look at each. This post reframes coding evaluation through that lens, then gets down to what teams should actually do.

The timing makes it matter. Coding models ship almost weekly, and teams increasingly make real procurement calls off a single leaderboard delta. If that delta is noise, the decision is a coin flip wearing a lab coat.

The Two Axes — Signal and Noise

The framework from the Allen Institute for AI (Ai2) gives the cleanest definitions.

Signal is a benchmark's discriminatory power — how far apart it pushes the scores of a better model and a worse one. Ai2 measures it as the spread of models at a given scale.
Noise is the variability you get when you run the same model again, or across training checkpoints.

A good benchmark has a high ratio of the two — a high signal-to-noise ratio (SNR). The interesting part is that bigger is not better. In Ai2's experiments, using only the 16 highest-SNR MMLU subtasks beat using all 57, cutting error by 32%. Swapping the scoring metric from pass/fail to bits-per-byte pushed MBPP's SNR from 2.0 to 41.8.

Ai2 reports that this ratio predicts a benchmark's usefulness better than its raw size does — swapping to a higher-SNR metric lifted its "decision accuracy," the odds of correctly picking the better of two models, from 68% to 93%. A small, clean eval can be more trustworthy than a large, noisy one, which is not the intuition most leaderboards are built on.

Translated to practice, there is one rule — when you compare two scores, first ask whether the gap is larger than the noise. If rank 1 and rank 3 sit inside the noise band, that ranking is an illusion, not information.

Where the Noise in Coding Benchmarks Comes From

Noise in coding evals arrives from four directions.

1) Contamination. The OpenAI team showed that a model can reproduce the original gold patch or problem statement nearly verbatim from just the task_id. That is memorization of training data, not the capability to write software. The score rises; the signal does not.

2) Design flaws. The same post finds that more than 60% of the remaining unsolved SWE-bench Verified problems are not actually solvable as scored. Forty-nine tests reject functionally correct submissions, and twenty-six demand behavior that is nowhere in the spec. The model can be right and still score zero.

3) Harness and infrastructure differences. Anthropic measured that scores move without changing a single model weight — just by adjusting the run environment. On Terminal-Bench 2.0 the gap between the most- and least-resourced setups was 6 percentage points (p < 0.01), and the infrastructure error rate fell from 5.8% to 0.5%. What separates the ranks can be the container config, not the model.

4) Overfitting and wear-out. New benchmarks don't last long either. SWE-bench Pro, built to improve on Verified, was found in an independent audit to have broken tasks in 27.4% of cases by automated pipeline and 34.1% by human review. And the same model swings hard when only the benchmark changes — one reported comparison had a single model at roughly 81% on SWE-bench Verified and roughly 46% on the harder SWE-bench Pro.

The uncomfortable part is that these effects aren't monotonic. Anthropic found that modest changes in resource headroom stayed within the noise (statistically insignificant, p = 0.40), while the extremes diverged sharply — and on SWE-bench, giving a task 5x the RAM moved the score only about 1.54 points. The same knob is negligible in one range and decisive in another, which is exactly how a harness gap hides in plain sight.

What Teams Should Actually Do

A leaderboard may be signal to the people training models, but to a team choosing an assistant it is mostly noise. The practical guidance:

Evaluate on your own tasks. Build a small private eval set from 30–50 of your team's real PRs and issues. "Pass rate on our codebase" is a far stronger signal than rank 1 on a public board.
Read confidence intervals, not ranks. Anthropic's advice is explicit — treat leaderboard gaps below 3 percentage points with skepticism until the eval config is documented and matched.
Prefer contamination-resistant designs. A rolling time cutoff (LiveCodeBench) or a private held-out test set (GAIA) structurally blocks memorization scores.
Pin and document the harness. A comparison that doesn't state prompts, retries, timeouts, and resource limits is comparing apples to oranges.
If you publish an eval, calibrate it. Anthropic's suggestion is to specify both a resource floor and ceiling per task and tune them so the scores at each end fall within noise of one another — otherwise the config silently becomes part of the score.
Triangulate across types. Read a static benchmark, a human-preference arena, and an agentic suite together, so the noise on any one axis cancels out.
Run each candidate more than once. A single pass/fail run hides the variance; three to five runs per task show you the spread — the very noise you're trying to see through.
Watch for saturation. When the top models cluster within a couple of points, the benchmark has stopped discriminating — low signal, and the order among them is close to random.

Concretely: freeze one scaffold — same prompt, same retry policy, same timeouts — run each model on your private task set several times, and report the mean with its spread. If two models' spreads overlap, you don't have a winner; you have a tie, and you should decide on price, latency, or licensing instead.

All of this converges on the eval-first principle — the eval set comes before the model (or the assistant), the same order laid out in the AI model development lifecycle post. Once the eval exists, choosing a model stops being taste and becomes an experiment.

Closing

Separating signal from noise is, in the end, a matter of humility. The last couple of decimal places at the top of a leaderboard are mostly marketing; the real signal is the difference that reproduces on your own tasks. OpenAI started from contamination and scoring flaws, Anthropic from infrastructure variance, Ai2 from statistical discrimination — and they arrive at the same place: don't trust a single number, ask whether that number is bigger than the noise. For a team choosing a coding assistant, there is little advice more practical than that.

None of this means benchmarks are useless — they are indispensable to the people training the models. It means a leaderboard is the start of a question, not the answer to it.