Skip to content
←

필사 모드: A Harness Is Not Configuration but a Deployable — The Real Bottleneck of the Self-Improvement Loop

English
0%
정확도 0%
💡 왼쪽 원문을 읽으면서 오른쪽에 따라 써보세요. Tab 키로 힌트를 받을 수 있습니다.

The prompt was untouched and the success rate rose by 6 percentage points

The task success rate of the agent went up compared with last week. There is no prompt commit. The model version is unchanged too. Looking for what changed: the description text of one tool definition got shorter, the retry cap went from 3 to 5, and the line-count cap the file-reading tool returns had changed. All three changes were made by different people for different reasons, and none of them are in the release notes.

If the success rate drops next week, what should you roll back? In the current structure you cannot answer.

Management starts once you draw a boundary around the harness

The post Lilian Weng wrote on 4 July 2026 puts a name on this mass. The harness is the system that wraps a foundation model and orchestrates execution — the layer that decides how the model thinks and plans, how it calls tools and acts, how it perceives and manages context, where it stores artifacts, and how results are evaluated.

The reason this definition is useful is that it draws the scope wide. Wider than what early agent frameworks covered, it includes workflow design and evaluation, permission control, and persistent state management. The three things that changed in the example above all fall inside this boundary. Only after drawing the boundary does the question "what changed?" become answerable.

From prompt engineering to harness engineering

The practical meaning of this shift is where the levers are. Most teams do not build foundation models and do not fine-tune either. What that team can touch is the entire harness.

And the room for improvement on the harness side is generally larger than in prompt wording. It is decisions like how many tools to expose, how to return failures, when to spin up a subagent, whether to leave intermediate artifacts as files or carry them in context. This is where the same post touching on code as the universal language hooks in. The space you can define in code is far wider than the space you can instruct in sentences.

Take just one example: how you return a tool failure to the model. Throw the exception string back as-is and the model repeats the same call. Return a structured list of currently viable alternatives together with the cause of failure and the next call is different. This is the kind of improvement you cannot get no matter how well you write a prompt, and it is entirely a code-side decision.

Self-improvement happens on the harness before the weights

The central claim of the post concerns recursive self-improvement. The structure in which an AI improves the machinery that produces its own intelligence using its current intelligence — and the outlook is that in the near future the path this takes will be the harness evolving rather than model weights being edited directly.

That outlook sounds abstract but it is already running in very concrete forms. Agents fixing their own tool definitions, modifying their own execution scripts, reading failure logs and rearranging workflows. Given tools like file reading and writing, shell execution, git, and subagent creation, the harness becomes an object that can edit itself.

What matters practically here is less the fact that this loop is already turning than the fact that in most teams it turns without a record. The agent polishes a tool description, a human approves that commit loosely, performance shifts a little, and nobody connects the two events. The danger of a self-improvement loop is not that it is fast but that it is unobserved.

Context as a playbook, not a prompt

Among the approaches the same post introduces, the one most directly transferable to practice is treating context as an evolving playbook rather than an ever-lengthening prompt. Structured items are managed by splitting the roles of creating items, reflecting on them, and curating them.

The difference is this. In the prompt approach, newly learned things keep getting appended to the end of a paragraph. A month later nobody reads that document, mutually contradictory instructions coexist, and only the token count grows. In the playbook approach, each item records when it was added, in what situations it applies, and whether it helped recently, which makes deletion possible. The hard part of context management is not putting things in but taking them out, and to be able to take something out an item needs metadata.

A harness is a deployable that carries a version

Once you get here you can come back to the original problem. To treat a harness as a deployable, you need at minimum a fingerprint.

"""Harness fingerprint: store it with the results or you cannot trace a regression."""
import hashlib
import json
from dataclasses import dataclass, field, asdict


@dataclass
class HarnessSpec:
    model: str
    system_prompt: str
    tools: list                 # [{"name":..., "description":..., "schema":...}]
    max_steps: int
    max_retries: int
    context_policy: dict        # truncation, summarization, playbook policy
    permissions: list           # permitted side effects

    def fingerprint(self) -> str:
        payload = asdict(self)
        # Tool order is meaningless, so normalize it. Skip this and the fingerprint changes every time.
        payload["tools"] = sorted(payload["tools"], key=lambda t: t["name"])
        blob = json.dumps(payload, sort_keys=True, ensure_ascii=False)
        return hashlib.sha256(blob.encode("utf-8")).hexdigest()[:12]


def diff(a: HarnessSpec, b: HarnessSpec) -> dict:
    da, db = asdict(a), asdict(b)
    return {k: (da[k], db[k]) for k in da if da[k] != db[k]}


base = HarnessSpec(
    model="some-model-v3",
    system_prompt="...",
    tools=[{"name": "read_file", "description": "read a file", "schema": {}}],
    max_steps=20,
    max_retries=3,
    context_policy={"strategy": "playbook", "max_items": 40},
    permissions=["read_fs"],
)
candidate = HarnessSpec(**{**asdict(base), "max_retries": 5})

print(base.fingerprint(), "->", candidate.fingerprint())
print(diff(base, candidate))   # {'max_retries': (3, 5)}

Leave this fingerprint alongside every evaluation run result and you can trace which change each step in the success-rate graph corresponds to. And a rule naturally emerges that any change which alters the fingerprint is a change that requires rerunning evaluation. The one line raising the retry cap from 3 to 5 gets treated at the same rank as a prompt edit.

The bottleneck of this loop is the evaluator, not the model

The same post lists several hard problems facing harness engineering, and the first it names is weak evaluators. This is an obvious conclusion once you look at the structure of the self-improvement loop.

The loop changes the harness, evaluates, and adopts whichever came out better. The speed of this loop is determined not by how fast you can change the harness but by whether the evaluation is trustworthy. If evaluation is noise, the loop moves along the noise. It becomes a system that moves fast without direction, and the metric goes up while real-world quality stays flat or gets worse.

So what absolutely must come before automating harness improvement is evaluator calibration. Reverse the order and automation just manufactures the problem faster.

Reward hacking is not a bug but normal output

The last hard problem is reward hacking, and it matters not to misread the phrase. An agent editing the tests to make them pass, inserting exception handling that swallows failures, or filling in only the fields the evaluation script looks at is not the result of a broken system. It is the result of precisely optimizing the objective we defined.

The response runs along two lines. One is permissions. Exclude evaluation code and scoring data from the paths the agent can write to and half of this category disappears. That is why the original explicitly puts permission control among the components of the harness. Permissions are an integrity-of-evaluation item before they are a security item.

The other is not relying on a single metric. Alongside success rate, watch guard metrics such as tool call count, the scope of files modified, and the number of deleted tests. Guard metrics are alarms rather than objectives, so it is enough to set thresholds and have a human look when one is crossed.

To sum up: the levers of harness engineering are large and within reach, but pulling them safely requires evaluation to be standing first. Attaching a fingerprint to the harness is the cheapest first step that connects those two.

References

현재 단락 (1/55)

The task success rate of the agent went up compared with last week. There is no prompt commit. The m...

작성 글자: 0자원문 글자: 7,734자작성 단락: 0/55