필사 모드: Coding Benchmarks Are Misaligned with the Agent Era — Three Reasons Leaderboards Compare Agents Wrong

• Introduction — Measuring Models, Comparing Systems
• A Coding Agent Is a System, Not a Model
• Three Ways the Alignment Breaks
• What Better Evaluation Looks Like
• Closing
• References

Introduction — Measuring Models, Comparing Systems

Coding agents have become a primary way many teams ship software. Yet the yardstick we rank them with still belongs to the era when we measured a single model. In Separating signal from noise in coding evaluations we watched leaderboards wobble under contamination and infrastructure noise; in the UniClawBench piece we saw one concrete answer for what a better agent benchmark could look like. The paper here answers the question sitting between them — why the mismatch exists, and what is structurally broken.

The position paper "Position: Coding Benchmarks Are Misaligned with Agentic Software Engineering" by a Tessl team (Maria I. Gorinova et al.), posted to arXiv on 16 June 2026, argues exactly what its title says: today's coding benchmarks are fundamentally misaligned with agentic software engineering.

The root of the mismatch is simple. Current benchmarks were designed in the pre-agent era to evaluate individual models. So they produce a single end-to-end score against one reference solution, with no signal about the intermediate steps. The trouble is that we now use that yardstick to compare systems, not models.

A Coding Agent Is a System, Not a Model

Here is the paper's central reframing. A coding agent in practice is not one model but a system harness — a composite of models, harnesses, contexts, environments, and feedback signals.

score = f(model, harness, context, environment, feedback)

The decisive claim about this picture: changing any one of these components can move the score by a margin comparable to the gap between adjacent model generations. In other words, adjusting the harness or the context alone can produce a score swing that looks like you upgraded to a whole new model generation.

It helps to make those components concrete. For a coding agent, the harness is the retry and planning loop; the context is what retrieval and prompt assembly put in front of the model; the environment is the container, tools, and test runner; and the feedback is what comes back from executing code. Each is a design decision, and each is folded into the same single score.

That puts a single leaderboard number in an awkward spot. When model A beats model B, the number alone cannot tell you whether A is the better model or whether A simply wore a better harness. The empirical backing is in the previous post: Anthropic moved scores using only container settings, without touching a single weight.

Three Ways the Alignment Breaks

The paper distills the misalignment into three concrete symptoms.

1) It conflates the model with the harness. Benchmark scores blend the model and the rest of the harness into one lump, so you cannot tell whether an improvement came from a better model or from better system design. When a vendor says "the new model gained a few points," much of that gain may actually be harness work.

2) A single reference penalizes valid alternatives. Grading against one reference solution marks equally correct alternatives as wrong. Software has many right answers; a functionally correct approach still scores zero if grading is tied to one gold patch's diff or tests. (OpenAI's observation cited in the earlier post — that over 60% of SWE-bench Verified's unresolved tasks were ungradeable by construction — is an extreme case of the same problem.)

3) There is no component-level signal. Because there is no signal at the level of individual harness components, the end-to-end system score is hard to iterate on. You get one number, and it does not say whether the bottleneck is context management, tool calls, or the feedback loop. For anyone building systems this is the deepest problem — you cannot fix what you cannot localize.

These three are not independent — they compound. Conflation hides which component earned the score; single-reference grading hides behavior that was actually correct; and the missing component signal means that even when you suspect the harness, you cannot confirm it. Stacked together, they turn the end-to-end number into something you can rank with but cannot learn from.

What Better Evaluation Looks Like

One caveat up front: the paper's abstract reframes the problem rather than prescribing a specific replacement benchmark. So what follows is the direction its frame points toward — and what designs like UniClawBench already implement in part. The three directions simply invert the three failure modes.

• Component-level. Instrument each component and vary one at a time (ablation), then report which part moved the number. Instead of a single end-to-end figure, the contribution of each layer of the harness should be visible.
• Multi-reference. Rather than string-matching one gold patch, grade on outcomes and behavior. UniClawBench's step-level checkpoints inside a live container — checking whether the thing actually happened — are a concrete instance of this direction.
• System-aware. Fix and document the harness, then compare systems as systems. It means dropping the illusion that a single number is "the model's ability."

In short, a good agent evaluation should tell you what (which component) moved and why (through which behavior). A single rank tells you neither.

Practically, most public leaderboards give you none of the three. So the burden shifts to you: pin your own harness, grade on your own outcomes, and ablate your own components on your own tasks. That is more work than reading a rank off a table — but it is the only version of the number that tells you anything about your system.

Closing

It is worth remembering this is a position paper. It is not a new benchmark or new data — it is a lens. So use it as a lens, not as a result to cite. That lens, though, is fairly sharp.

Compressed to one sentence: when we compare two agents by a single leaderboard number, we are really comparing two whole systems while pretending to compare two models. The useful question is therefore not "who is number one" but "which component created the gap, and can I reproduce it on my own harness." Until component-level, multi-reference, and system-aware evaluation are in place, the decimals on an agent leaderboard are mostly configuration, not signal.

References

• Position: Coding Benchmarks Are Misaligned with Agentic Software Engineering (arXiv)
• Separating signal from noise in coding evaluations (related post)
• The 2026 agent benchmark seen through UniClawBench (related post)