Split View: 추론 강도는 모델 선택이 아니라 요청 단위 배포 파라미터입니다

Reasoning Effort Is Not a Model Choice but a Per-Request Deployment Parameter

• Same model, three scores on the page
• What the ARC results page actually tells you
• What effort buys differs threefold with task difficulty
• What deciding per request means
• The ladder only holds when there is a verifier
• Substitute signals when you cannot build a verifier
• What this page cannot tell you
• What you have to record to treat it as a deployment parameter
• References

Same model, three scores on the page

A new model comes out and you go to check the benchmark results. But there is not one score. There are three. Which number should you post in Slack?

This situation is now closer to the default than to the exception. As models with adjustable reasoning effort proliferate, a single model has started to be published as a curve rather than a point. And this change is not a change in publication format but a change in deployment design.

It used to be that choosing the model was the end of the decision. Once you decided which model to use, performance and cost were fixed together. Now one decision remains even after choosing the model, and that decision swings performance and cost hard in both directions. Hard-coding it as a constant in a config file means giving up that whole swing.

What the ARC results page actually tells you

Let us look at the ARC Prize results page for DeepSeek V4 Flash 0731. The model was released on 31 July 2026, and three levels of reasoning effort are each measured.

Cost is listed only on a max-effort basis: $0.02 per task on the ARC-AGI-1 semi-private evaluation, and $0.04 per task on ARC-AGI-2.

What effort buys differs threefold with task difficulty

There is one thing computed directly off this table: the gain from raising effort from lowest to maximum.

On ARC-AGI-1 it goes from 84.0 to 89.0, a rise of 5.0 percentage points. On ARC-AGI-2 it goes from 46.0 to 61.4, a rise of 15.4 percentage points. The same dial turned the same amount earned more than three times as much on the harder side.

Reading it the other way round is more useful. Using maximum effort on easy tasks is mostly waste. On ARC-AGI-1 the lowest effort already delivers about 94 percent of what maximum effort does. Conversely, running the hard tasks at lowest effort is a choice that throws away a lot of performance.

So the question "what effort should we use this model at?" is malformed from the start. Effort is not a property of the model but a value to be set to match the properties of the task.

What deciding per request means

In practice this conclusion becomes routing logic, not a line in a config file. For each incoming request you estimate difficulty and choose an effort level.

The difficulty estimate does not have to be perfect. For most services even very coarse signals are enough: input length, the number of steps required, the failure rate on past requests of the same type, the urgency the user stated.

That said, trying to get this estimate right in one shot hits a wall soon. The moment you attach a classifier that predicts difficulty, that classifier becomes something to maintain, and it has to be retrained whenever the distribution shifts. The sturdier method is not to estimate but to escalate when you were wrong. Using observation instead of prediction almost always lasts longer.

The ladder only holds when there is a verifier

"""Escalation ladder: try cheap first, and raise the level once it is confirmed wrong."""
from typing import Callable, Sequence

LADDER = ("low", "high", "max")


def solve_with_escalation(
    task,
    run: Callable[[object, str], object],       # (task, effort) -> answer
    verify: Callable[[object, object], bool],   # (task, answer) -> pass or not
    ladder: Sequence[str] = LADDER,
):
    """Raise effort until the verifier passes it. Also return the effort that was used."""
    attempts = []
    for effort in ladder:
        answer = run(task, effort)
        ok = verify(task, answer)
        attempts.append((effort, ok))
        if ok:
            return answer, effort, attempts
    return answer, ladder[-1], attempts   # if it fails all the way, return the final attempt


def expected_cost(p_pass_by_effort: dict, unit_cost: dict, ladder=LADDER) -> float:
    """Expected cost of the ladder. The higher the pass rate at low levels, the lower the average."""
    total, reach = 0.0, 1.0
    for effort in ladder:
        total += reach * unit_cost[effort]
        reach *= 1 - p_pass_by_effort[effort]
    return total


p = {"low": 0.80, "high": 0.60, "max": 0.50}     # pass rate given that this level was reached
cost = {"low": 0.004, "high": 0.010, "max": 0.020}
print(round(expected_cost(p, cost), 5))          # compare against using max effort only

The premise of this structure is verify. If verification is impossible the ladder does not hold, and the only remaining option is to run at high effort from the start. So if you want to optimize effort, the first question to ask is not "how do we choose the effort level" but is there a cheap way to find out that we were wrong.

One more thing to watch is latency. The ladder lowers average cost but raises worst-case latency. A request that walked all three levels finishes later than a request run at maximum effort in one go. So on a synchronous path where a user is waiting, you need a design that cuts the ladder to two levels, or streams the low level first and replaces it with the escalated result. For batch work you do not need to worry about this.

Substitute signals when you cannot build a verifier

Even without a complete verifier, partial signals can usually be built.

For code generation, compilation and test execution are the verifier as-is. For structured output, schema validation and referential integrity checks catch a good portion. For retrieval-grounded answers, you can confirm by string comparison whether a quoted sentence actually exists in the source.

For free-form tasks where even these are unavailable, look for the signal in the process rather than the result. Run the same input twice at low effort, and if the answers differ substantially, that request is likely a hard one for this model. In the range where running twice is cheaper than stepping up one level, this method is practical.

If you hand the judgement of "the answers differ" back to a model, the cost advantage disappears. Cheap comparisons are often enough — cosine distance between sentence embeddings, or set comparison over the key numbers and proper nouns extracted from the answers. A verifier does not need to be accurate; it needs to be cheap and wrong in only one direction. It is fine for it to miss failures, but it should rarely judge a correct answer to be wrong, or the ladder does not run without waste.

What this page cannot tell you

There is a part worth flagging honestly. The table above has three rows of per-effort scores, but cost has only one row, on a max-effort basis. The per-task cost at lower effort levels is not confirmable from this page.

So marginal accuracy per cost cannot be fully computed from this data alone. The unit prices in the code above are only an example to show the structure, and the real values have to be measured by each team on its own traffic. And this is less a defect of this page than the current publication convention across the industry. Per-effort scores have multiplied while per-effort costs still do not come out alongside them.

Measuring is not hard. Run 100 items from your own evaluation set at each of the three efforts, record accuracy together with actually billed tokens, and the curve appears. This experiment takes half a day and becomes the basis for every effort decision afterwards.

One more thing to keep in mind is that the difficulty distribution of a benchmark differs from the difficulty distribution of your traffic. ARC tasks are a collection of problems deliberately designed to be hard, whereas real service traffic is generally a long-tail distribution overwhelmingly dominated by easy requests. So the lesson to take from the table above is not "we should use maximum effort too" but the fact itself that the dial is worth different amounts in different difficulty bands.

What you have to record to treat it as a deployment parameter

The last piece is logging. Once you start setting effort per request, the moment you fail to log the effort, every metric becomes uninterpretable.

Store at minimum these four things alongside the response: the effort actually used, the number of attempts on the ladder, the verification result of each attempt, and the total tokens consumed. Add which difficulty band the request was classified into and you can later tune the routing rules with data. With this you can immediately separate the cause in a situation like "accuracy went up versus last week but so did cost." Without it, you will never know whether it was the model, the routing, or a shift in traffic composition.

Returning to the opening question: the number to post in Slack is not one of the three. Posting all three and noting which one our tasks resemble is the right answer.

References

• DeepSeek V4 Flash 0731 — ARC Prize results page — the three rows of per-effort scores, the per-task cost on a max-effort basis, and the model release date are what is on this page. The 5.0 and 15.4 percentage point figures in the body are values obtained by subtraction from that table.
• ARC Prize — has the distinction between semi-private and public evaluation and an explanation of the evaluation method.
• The pass rates and unit prices in the ladder code are example numbers for explaining the structure, not values taken from the ARC page.
• Related post on this blog: The claim of 100x cheaper is true only when the task was narrowed