Skip to content
Published on

What Is Harness Engineering — The Model Is a Fixed Input; What You Ship Is Everything Around It

🇰🇷KO🇺🇸EN🇯🇵JA🇨🇳ZH
Split View필사 모드
Share
Authors

Same model, so why the different results

Suppose two teams automate the same kind of task with the same foundation model. One team's agent takes an issue and produces a patch that passes the tests; the other team's agent digs through files on a similar issue and stalls halfway. This contrast is a constructed example for the sake of explanation, but the shape of it will feel familiar to any team that has operated agents. If the model is the same, the difference came from outside the model.

Write down what sits outside the model and the list is longer than you expect. Which tools were exposed, and how many. What the model gets back when a tool fails. How many retries are allowed, and on what grounds the run stops. What goes into the context window each turn, and what gets dropped. Which paths the agent is allowed to write. And who judges whether the result was good, by what standard. That entire list exists as code, gets deployed, and moves the task success rate.

The harness: the whole execution system around the model

The post Lilian Weng wrote in July 2026 gave this list a name. The harness is the system that wraps a foundation model and orchestrates its execution — the layer that decides how the model thinks and plans, how it calls tools and acts, how it perceives and manages context, where it stores artifacts, and how results are evaluated.

This series divides what sits inside that boundary into six knobs.

  • The tool surface — the set of tools exposed to the model, with their schemas. How many, under what names and descriptions.
  • The failure return format — when a tool fails, does the model get the raw exception string, or a structured cause plus alternatives.
  • The loop — retry caps, stopping conditions, escalation when stuck.
  • The context policy — what goes in, what gets dropped, when to summarize.
  • Permissions — the range the agent can read and write. In particular, whether the evaluation code sits inside it.
  • The evaluator — the grader that judges results. This one sets the ceiling for the whole system.

There is one premise. For most teams, the model is a fixed input. A team that neither builds nor fine-tunes a foundation model can touch exactly these six things — and these six are plenty.

What the name "prompt engineering" hides

The long-standing name for this work was prompt engineering. The name is not wrong, but it badly undersells the scope. The prompt appears nowhere in the six knobs as a standalone item. The system prompt is one piece of the context policy, and tool description copy is part of the tool surface. The space you can define in code is far larger than the space you can instruct in sentences, and most of the decisions that move success rates live in the former.

The agent-building guide Anthropic published in December 2024 makes this point from the tool side. Just as humans get a UI, agents get tool definitions as their interface, so tool design deserves as much care as the prompt. The same post notes that in their own SWE-bench work, more time went into optimizing the tools than the overall prompt.

The failure return format is the same kind of example. In a harness that returns the raw exception string, the model tends to repeat the same call; in a harness that returns the cause of failure plus the alternatives currently available, the next call is different. No amount of prompt polishing buys you this difference. It is entirely a code-side decision.

Workflow or agent, the harness is there either way

The same Anthropic post splits agentic systems in two. Workflows arrange LLMs and tools along predefined code paths; agents let the model decide its own next actions and tool use. And it recommends starting with the simplest solution and adding complexity only when needed. For many problems, a single call with retrieval and examples is genuinely enough.

Reread that distinction from the harness point of view and it comes out like this. A workflow still needs the tool surface, the failure return format, permissions, and an evaluator. As you move toward agents, control of the loop shifts to the model, so the weight of retry caps, stopping conditions, and context policy grows sharply. Either way, what you deploy is a harness — and management begins when you can say what you deployed.

The knobs do not turn independently

Write the six knobs as code and you get this series in outline.

harness = {
    "tools": ["read_file", "write_file", "run_tests"],      # Part 3: the tool surface
    "on_error": "cause_plus_alternatives",                  # Part 3: how to return failure
    "loop": {"max_retries": 3, "stop": "goal_check"},       # Part 4: loops and stopping
    "context": {"policy": "playbook", "budget": 12000},     # Part 2: the context budget
    "permissions": {"write": ["src/"], "deny": ["eval/"]},  # Part 6: reward hacking and permissions
    "evaluator": "rubric_v3",                               # Part 5: the evaluator bottleneck
}

The thing to watch is that the knobs are entangled. Raising the retry cap is entangled with the failure return format: retries without a cause just repeat the same failure at higher cost. Adding tools is entangled with the context budget: tool schemas eat tokens too. And every knob is entangled with the evaluator, because the evaluator is what judges which combination is better — with a weak evaluator, the comparison itself does not hold. So this series takes the knobs one at a time but keeps returning to the same question: how do you know this change made things better?

Practice it yourself

This blog ships a harness engineering RPG that turns this into a game: you assemble the six knobs you just saw and clear 27 scenarios. Tier 1, "Observation and fingerprints", corresponds to this post. If you want to build up the prompt-side fundamentals, there is also a prompt engineering practice tool.

References

Discuss on Twitter • View on GitHub