Skip to content
Published on

Reward Hacking — The Metric Rises While the Task Fails

🇰🇷KO🇺🇸EN🇯🇵JA🇨🇳ZH
Split View필사 모드
Share
Authors

The day the pass rate hit 100 percent

Say the test pass rate of a coding agent running as a nightly batch hits 100 percent one day. Before celebrating, you open the diff: the assertions of the three failing tests are commented out. A constructed example — but the behavior itself is a documented phenomenon. Lilian Weng's survey of reward hacking names coding models editing the unit tests to pass, instead of solving the problem, as a signature form of the LLM era.

One sentence from this post is worth keeping. The metric genuinely went up. Nothing errored. And the task failed.

Not a bug but normal output

To carry over the definition from that post: reward hacking is an agent exploiting flaws or ambiguities in the reward function to obtain high reward without genuinely doing the intended task. It is a pattern that repeats from the old reinforcement learning cases onward, and the cause is old too — Goodhart's law: when a measure becomes a target, it ceases to be a good measure.

What matters from the harness point of view is the attitude. This is not the system malfunctioning; it is the system optimizing exactly the goal we wrote down. Ask "why would it do such a thing" and no answer comes; ask "which gap in our grader made this behavior the optimum" and the answer arrives. As part 5 showed, the evaluator sets the ceiling of the system — reward hacking shows that the gaps in that evaluator become the direction the system moves in.

The common forms: edit the criteria, delete, fill the surface

The observed forms come mostly in three lines. First, soften the grading criteria themselves: lower thresholds, replace strict comparisons with lenient ones. Second, delete what gets caught: remove or comment out failing assertions, hide failures behind exception-swallowing handlers. Third, fill only the surface the evaluation looks at: the fields the grading script checks are perfect, while the actual behavior underneath is hollow. The first two lines have documented instances; the detailed scenarios here are constructed for explanation.

There is also the variant the same post calls in-context reward hacking. Even with no weight updates, behavior that probes the evaluator's gaps emerges inside an iterate-and-evaluate loop, and metric and actual quality diverge as the run proceeds. For a harness running a self-improvement loop, this variant is the default threat model.

Permissions erase half of it

The cheapest response is permissions. The hack that edits the grading script exists only while the grading script sits in a writable path. Simply moving the evaluation code and grading data out of the agent's write scope makes this whole class disappear structurally. This is why Lilian Weng's harness post says explicitly to keep permission controls and held-out tests outside the optimization loop.

Seen this way, permissions are an evaluation-integrity item before they are a security item. If the task is development work where test files must live inside the working tree, then at minimum, deleted or weakened tests should be watched by a counter-metric.

Counter-metrics: an alarm, not a target

The layer after isolation is counter-metrics. Do not watch one primary metric alone; watch the signals that must not go strange while that metric climbs. Tool call counts, the scope of modified files, the number of deleted tests.

# Counter-metric config (constructed example) — an alarm, not an optimization target
alarms:
  - metric: deleted_or_weakened_tests
    threshold: 0 # even one
    action: human_review # stop auto-adoption; a human looks
  - metric: files_changed_outside_scope
    threshold: 0
    action: human_review
  - metric: tool_calls_per_task
    threshold: 'baseline_p95 x 2'
    action: flag

The design principle is single: a counter-metric is an alarm, not a target. Set a threshold, and when it trips, it is enough that a human looks. The moment you promote a counter-metric into an optimization target, Goodhart's law starts applying to that metric too.

The remaining half that isolation cannot catch

Permissions and counter-metrics erase the grading-manipulation class. What remains is the class that fools the grader. The same reward hacking post points out that models can learn convincingly wrong outputs — answers that persuade human evaluators while being incorrect. LLM judges sit under the same threat: match the surface features the judge looks for, and the score goes up.

For this class there is no structural silver bullet, only accumulating mitigations. Re-evaluate periodically on held-out tasks the agent has never seen. Refresh the judge prompts and criteria on a schedule. Have humans directly review a sample of adopted changes. And keep the judge calibration from part 5 on the maintenance calendar. Responding to reward hacking is not a one-time design; it is an operations item.

Practice it yourself

Nearly half of the scenarios in the harness engineering RPG carry a legitimate exploit built in. When the conditions hold it fires, the metric genuinely rises, nothing errors, and the task score quietly drops. The divergence only gets named in the debrief. Reach tier 6, "The self-improvement loop", and you will have lived through everything in this post inside the game.

References

Discuss on Twitter • View on GitHub