Skip to content
Published on

The Evaluator Bottleneck — A Weak Grader Caps the Whole System

🇰🇷KO🇺🇸EN🇯🇵JA🇨🇳ZH
Split View필사 모드
Share
Authors

You fix the harness and the score does not move

Say you refine the tool surface, structure the failure returns, and turn the context into a playbook — and the success-rate graph stays flat. A constructed example, but this flatness has one common cause: not that there was no improvement, but that the current evaluator cannot see it. Under a grader that counts anything that does not crash as a pass, every change that raises the quality of the result is invisible to the score.

What makes this dangerous is that the conclusion comes out backwards. The true state is "we cannot measure what improved", but the graph reads as "fixing the harness does not help".

You cannot select for a quality you cannot measure

This is the principle the series keeps returning to. Harness improvement is ultimately a selection loop: make a change, evaluate it, adopt the better side. The only qualities this loop can select for are the ones the evaluator can distinguish. On any axis the evaluator cannot tell apart, even excellent changes get adopted at chance. A weak grader is therefore not a minor shortfall; it is the ceiling of the entire system.

Lilian Weng's harness post also puts weak evaluators first in its list of hard problems for harness engineering: many tasks lack a fast and precise verifier, and some qualities — research taste, long-term value — are hard to build a verifier for at all. The diagnosis that the bottleneck sits in evaluation rather than elsewhere is close to a shared conclusion among the people actually pushing this field.

The evaluator ladder: from smoke to panel

An evaluator is not a yes/no matter but a matter of strength. This series organizes it as a five-rung ladder.

  • Smoke — checks only that the run finished. Not what it did. Result quality, scope of change, and regressions are all blind spots.
  • Unit-test pass rate — you get one number. It means something only while the test files are honest, and quality no test covers stays a blind spot.
  • Rubric grading — write down in advance what counts as good and grade against it. Noise drops a lot.
  • Rubric plus sealed grading data — the rubric and answer set are neither readable nor writable by the agent.
  • Rubric plus a counter-metric panel — not just the success rate: tool calls, modification scope, and deleted tests are watched together.

It gets more expensive as you climb. The point is not to always use the top rung, but to know which rung you stand on and what its blind spots are. Carefully fixing a harness under a smoke evaluator is cooking with a scale that has no markings.

The rubric: write down in advance what counts as good

The middle rung is where practice gets the best return. Anthropic's multi-agent system retrospective describes giving an LLM judge a rubric and having it output a 0.0–1.0 score and a pass-fail grade. The criteria: factual accuracy, citation accuracy, completeness, source quality, tool efficiency. And the retrospective adds that human testing caught the subtle failures automated grading missed.

# Rubric for a code-review summary agent (constructed excerpt)
- id: grounded
  weight: 3
  pass: 'Every finding carries a file path and line-level evidence'
- id: scope
  weight: 2
  pass: 'No files outside the requested change scope are mentioned'
- id: actionable
  weight: 2
  pass: 'There is a next step the reviewer can execute immediately'

The value of a rubric comes before grading automation: it documents an agreement. If the team has not agreed on what a good result is, any grader you attach will surface that disagreement as noise.

Sealing and counters: putting the grading out of reach

From the fourth rung upward, the subject becomes the integrity of evaluation. If the grading criteria and answer data sit inside paths the agent can write, the cheapest way to raise the metric is no longer doing the task well — it is editing the grading. The fifth rung's counter-metrics watch whether other signals go strange while the success rate climbs. Both rungs connect directly to part 6 on reward hacking, which covers them in detail.

Calibrate the evaluator first

The conclusion is about order. If you want to automate harness improvement, evaluator calibration comes first. If the evaluation is noise, the loop follows the noise, and automation only makes that movement faster. A system whose metrics rise while real-use quality stays flat is usually the result of reversing this order.

The practice of calibration is covered elsewhere on this blog: measure agreement between human labels and judge verdicts, then revise the rubric using the disagreement cases, and repeat. In evaluation-driven development, the first thing to calibrate is the judge pairs well with this post.

Practice it yourself

Tier 5 of the harness engineering RPG is literally the title of this post: "The evaluator bottleneck". In the game, the evaluator is implemented as the rule that caps the task score, so you can watch how the same harness splits between a run under smoke and a run under a rubric.

References

Discuss on Twitter • View on GitHub