Skip to content
Published on

Harness Fingerprints and Versioning — Making Unrecorded Changes Traceable

🇰🇷KO🇺🇸EN🇯🇵JA🇨🇳ZH
Split View필사 모드
Share
Authors

Once more: six points and no commit

The earlier post this series grew out of opened with one constructed case. The success rate rose six percentage points over last week with no prompt commit and no model change — and it turned out one tool description line, the retry cap, and a file-read line cap had each been changed by different hands. If the success rate drops next week, what do you roll back? Without a record, there is no answer.

This post covers the minimal device that makes that incident answerable: the harness fingerprint. To say it upfront — the fingerprint is not something Lilian Weng's harness post proposed. It is an approach this blog constructed to carry the boundary that post drew into operations.

The harness is a deployable that carries a version

The premise of the fingerprint is a shift of viewpoint: see the harness not as a scattered pile of settings but as one deployable. A deployable has a version, and with versions you can ask what changed between two points in time. This is where Weng's definition earns its keep. Only with a boundary — the whole layer that orchestrates how the model thinks and acts, its context, its artifacts, its evaluation — can you answer "is this change a harness change" consistently. Polishing one tool description line and raising the retry cap both sit inside the boundary, so both are version-moving changes.

Building the fingerprint: normalization is half the work

The fingerprint itself is short: serialize the decisions that make up the harness and hash them. The hard half is normalization. Two harnesses with the same meaning must produce the same fingerprint, but a naive serialization changes the hash when nothing but the tool registration order changed. So sort the tool list by name, fix JSON keys in sorted order, and unify whitespace and field order that carry no meaning before serializing. Conversely, values that differ per run — timestamps, run IDs — must not enter the fingerprint. The moment they do, every run becomes its own incomparable harness.

What goes in, and what stays out

What goes in is every decision that changes the agent's behavior: the model identifier, the system prompt, the full tool schemas, retry caps and stopping conditions, the context policy, the permission scope, and the version of the evaluator. The evaluator goes in for exactly the reasons parts 5 and 6 showed: when grading criteria change, the same harness earns a different score, so grading changes are tracked changes too.

What stays out comes in two kinds. One is run attributes: task inputs, execution time, random seeds are metadata of a run, not of the harness. The other matters more. Logging and observability settings stay out of the fingerprint, because what you can see does not change what the agent does. Put observability into the fingerprint and results with identical behavior shatter into incomparable fragments. This rule is implemented as-is in this blog's harness RPG: observation equipment unlocks as you progress, and the fingerprint does not move.

Results must carry the fingerprint for comparisons to hold

The use of the fingerprint is stamping evaluation results.

{
  "run_id": "2026-08-12T03:14:07Z-a41",
  "harness_fingerprint": "3f9c1d2ab714",
  "suite": "issues-50",
  "success_rate": 0.62,
  "counter_metrics": { "deleted_tests": 0, "avg_tool_calls": 11.4 }
}

Once this line exists, two rules follow. First, comparisons hold only between equal fingerprints. Placing two success rates with different fingerprints side by side is not a comparison but an experiment — and an experiment must be able to state its diff. Second, every change that moves the fingerprint is a change that reruns the evaluation. Editing one tool description line starts being treated at the same rank as a prompt overhaul, and that is precisely the intended effect.

Rollback and bisecting regressions

When a step appears in the success-rate graph, the fingerprint history builds the suspect list.

def regression_boundary(runs):
    """Find the fingerprint boundary where the success rate broke (constructed example)."""
    runs = sorted(runs, key=lambda r: r["ts"])
    for prev, cur in zip(runs, runs[1:]):
        if cur["success_rate"] < prev["success_rate"] - 0.03:  # noise margin
            return prev["harness_fingerprint"], cur["harness_fingerprint"]
    return None

# The spec diff between the two boundary fingerprints is the suspect list.
# With several suspects, bisect: revert half at a time.

The spec diff between the two fingerprints is the candidate set of changes. With one candidate, revert it and re-evaluate; with several, narrow by halves in the same manner as git bisect. Rollback is defined over the same history: redeploy the spec of the earlier fingerprint, confirm the fingerprint of what got deployed matches the target, and the rollback is done. The old "it was probably this configuration" becomes "the hash matches".

Practice it yourself

The first scenario of tier 1, "Observation and fingerprints", in the harness engineering RPG is exactly the incident at the top of this post. As you assemble a harness to reproduce last week's six points, you learn by hand which changes the fingerprint reacts to and which it ignores.

References

Discuss on Twitter • View on GitHub