필사 모드: LLM API Cost Optimization — Break-Even Math for Output Tokens, Prompt Caching and Routing

Introduction — the invoice arrives in tokens

Discussions about LLM cost usually start at "let us switch to a cheaper model" and end at "but quality will drop." Neither side has evidence, so nothing gets decided.

Yet this is one of the easier problems to build evidence for. The bill is token count times unit price, and both numbers are already in your logs. All you need is to check which term dominates the invoice, then compute what percentage each lever removes from that term.

This post does that calculation in order. It decomposes the cost structure first, walks the levers from largest saving to smallest, and gives a break-even formula for each. Prices differ per provider and change often, so everything stays a variable; the examples use the common ratio of 3 dollars per million input tokens and 15 dollars per million output tokens. Plug in your own rates and rerun.

The structure of cost — input and output do not cost the same

The first fact to establish is that input tokens and output tokens are not priced alike. Output is typically three to five times more expensive than input. The reason is structural: input is processed in parallel in a single prefill pass, while output is a sequential job in which every token has to traverse the whole model again.

This asymmetry breaks intuition often.

def daily_cost(reqs, in_tok, out_tok, price_in, price_out):
    """price_* is the dollar price per million tokens"""
    c_in = reqs * in_tok * price_in / 1e6
    c_out = reqs * out_tok * price_out / 1e6
    return c_in, c_out, c_in + c_out

c_in, c_out, total = daily_cost(
    reqs=100_000, in_tok=2_000, out_tok=500,
    price_in=3.0, price_out=15.0,
)
print(f"입력 {c_in:8.0f}  출력 {c_out:8.0f}  합계 {total:8.0f}  (일)")
print(f"출력 비중 {c_out / total:.1%},  월 {total * 30:,.0f}")
# 입력      600  출력      750  합계     1350  (일)
# 출력 비중 55.6%, 월 40,500

Output is a quarter of the token count of input and still costs more. Trimming 500 tokens off the prompt and trimming 125 tokens off the response have the same financial effect. Most teams spend their time on prompt dieting and leave output length alone.

So the first question is always this one: why are our outputs this long?

The biggest lever is output length

Outputs grow long for roughly three reasons. The model adds an introduction and a conclusion, it appends explanation nobody asked for, and it wraps a structured result in prose.

In order of impact, here is what to do.

First, change the output format from prose to structure. Ask for the same information as a JSON schema or an enum and the token count drops to a fraction. The difference between "tell me the classification with an explanation" and "output the label only" is 200 tokens versus 3. How to get that reliably is covered in structured output reliability.

Second, lower max_tokens to what you actually need. This is less a cost saving than an accident guard. A model falling into a repetition loop and generating up to the ceiling is not rare, and that single request costs as much as hundreds of normal ones.

Third, narrow the range where you use chain of thought. Step-by-step reasoning raises accuracy on hard problems but multiplies output tokens. Rather than applying it uniformly to every request, judge difficulty first and attach it only where it is needed.

There is one thing to be honest about here. There are tasks where shortening the output does hurt quality. Summarization and classification lose almost nothing when made terse, but cutting the reasoning trace out of a task that requires reasoning lowers accuracy. So output reduction is not something you do unconditionally, it is something you measure per task and then do. How to measure it comes later.

Prompt caching — it only catches prefixes

If your service prepends the same system prompt and few-shot examples to every request, the input cost of that section is, in principle, a repeated charge. Prompt caching removes the repetition.

There is one core constraint. The cache is only keyed on prefixes. Reuse extends from the front of the prompt up to the point where the tokens still match exactly, and the moment a single token differs, everything after it is recomputed. An identical paragraph sitting in the middle is not caught.

This constraint effectively dictates the order in which you lay out a prompt.

# bad order: zero cache hits
prompt = f"""오늘 날짜는 {today}입니다.
{SYSTEM_RULES}          # 8,000-token fixed instructions
{FEW_SHOT_EXAMPLES}     # 4,000-token fixed examples
사용자 질문: {question}"""

# good order: the leading 12,000 tokens hit the cache wholesale
prompt = f"""{SYSTEM_RULES}
{FEW_SHOT_EXAMPLES}
오늘 날짜는 {today}입니다.
사용자 질문: {question}"""

Put one line with the date at the very front and the twelve thousand tokens behind it get recomputed every time. Change the position alone and you save all of it. The principle is simple: unchanging things first, frequently changing things last.

The pricing structure is generally that a cache read costs about a tenth of the base input rate while a cache write costs a little more than the base rate. The exact multipliers and the cache lifetime have to be confirmed in the provider documentation. Under that structure, the break-even works out as follows.

def caching_breakeven(write_mult=1.25, read_mult=0.1):
    """how many reuses within the cache lifetime does it take to come out ahead"""
    # for n uses: write_mult + (n-1) * read_mult  vs  n * 1.0
    n = 1
    while write_mult + (n - 1) * read_mult >= n:
        n += 1
    return n

print(caching_breakeven())  # 2

def cached_input_cost(reqs, static_tok, dynamic_tok, price_in, hit_rate,
                      write_mult=1.25, read_mult=0.1):
    dyn = reqs * dynamic_tok * price_in / 1e6
    stat_tok = reqs * static_tok
    stat = (stat_tok * hit_rate * read_mult + stat_tok * (1 - hit_rate) * write_mult) * price_in / 1e6
    return dyn + stat

base = 100_000 * 2_000 * 3.0 / 1e6
cached = cached_input_cost(100_000, static_tok=1_600, dynamic_tok=400,
                           price_in=3.0, hit_rate=0.9)
print(f"캐시 없음 {base:.0f} → 적중률 90%에서 {cached:.0f} (일 {base - cached:.0f} 절감)")
# 캐시 없음 600 → 적중률 90%에서 223 (일 377 절감)

Two reuses is already ahead. Caching is therefore not something to deliberate about adopting, it is something to deliberate about raising the hit rate on. The usual culprits that kill the hit rate are timestamps, request IDs and user names near the front of the prompt, plus tool definitions serialized in a way that does not guarantee order.

RAG versus the whole context — where is the break-even

As context windows grew, "why not just put everything in" became an option. On cost alone, the math is simple.

def per_request_cost(tokens, price_in, cached=False, read_mult=0.1):
    mult = read_mult if cached else 1.0
    return tokens * price_in * mult / 1e6

DOC = 100_000     # whole-document tokens
K, CHUNK = 5, 400 # RAG: top 5 chunks

full_raw    = per_request_cost(DOC, 3.0)
full_cached = per_request_cost(DOC, 3.0, cached=True)
rag         = per_request_cost(K * CHUNK, 3.0)
print(f"전체 {full_raw:.3f} / 전체+캐싱 {full_cached:.3f} / RAG {rag:.3f} (요청당 달러)")
# 전체 0.300 / 전체+캐싱 0.030 / RAG 0.006

# break-even document size: even allowing for RAG retrieval overhead
print(f"RAG가 유리해지는 문서 크기: {K * CHUNK}토큰 초과")

The numbers say two things. First, RAG is 50 times cheaper than stuffing everything in, but if the document is static enough for caching to apply, that gap narrows to 5 times. Second, 5 times is still a large difference, but it is a difference you have to weigh against the cost of building and maintaining a RAG pipeline. On a service doing a thousand requests a day, a 0.024 dollar per request gap is a bit over 700 dollars a month. Engineer time costs more.

So I think this decision is better made on axes other than cost.

If the document does not fit in the context window, there is no option but RAG. If the document differs per user, the hit rate is low and stuffing everything in gets expensive. If latency matters, the time to first token behind a 100,000-token prefill becomes the problem. Conversely, if the document is small (tens of thousands of tokens), static, and the price of a retrieval miss is high, putting everything in is simpler and safer. And since models have been reported to lose track of information placed in the middle as the context gets longer, the assumption that "stuffing everything in always gives higher accuracy" is itself something to verify.

Model routing — putting the saving and the accuracy loss in one table

How much you save by sending easy requests to a small model is decided by two numbers: the delegation share and the price ratio.

def routing_saving(p_small, price_ratio):
    """p_small: share sent to the small model, price_ratio: small model rate / large model rate"""
    return p_small * (1 - price_ratio)

def cascade_saving(price_ratio, escalate_rate, judge_ratio=0.0):
    """answer with the small model first and escalate to the large one when it falls short. Escalations pay twice."""
    cost = price_ratio + judge_ratio + escalate_rate * 1.0
    return 1 - cost

print(f"라우팅   {routing_saving(0.7, 1/15):.1%}")            # 65.3%
print(f"캐스케이드 {cascade_saving(1/15, 0.25, 0.02):.1%}")   # 66.3%

# condition for a cascade to hold: escalation rate below (1 - price ratio - judging cost)
print(f"승급률 상한 {1 - 1/15 - 0.02:.1%}")                   # 91.3%

On cost alone both approaches leave a lot on the table. A cascade comes out ahead as long as the escalation rate stays under 91 percent, which means it is effectively always ahead. In other words, the bottleneck in this decision is not cost, it is quality.

So you must put this next value on the same screen as the saving rate.

def routed_accuracy(p_small, acc_small, acc_large):
    return p_small * acc_small + (1 - p_small) * acc_large

acc = routed_accuracy(0.7, acc_small=0.88, acc_large=0.94)
print(f"라우팅 후 정확도 {acc:.3f} (단일 대형 0.940, 손실 {0.94 - acc:.3f})")
# 라우팅 후 정확도 0.898 (단일 대형 0.940, 손실 0.042)

The 0.88 and 0.94 in there are example values I made up. In practice you have to measure each model on the distribution you are actually routing, and plug those in. There is one trap in that measurement. You need accuracy on "the requests you decided to send to the small model," not accuracy on the whole evaluation set. If the router is doing its job, the gap between the two models on that subset should be much smaller, and if the gap is unchanged, that is a signal the router is not distinguishing difficulty at all.

The additional thing to watch in a cascade is the reliability of the escalation judgment. If a wrong judgment fails to escalate a bad answer, that is not a saving, it is simply degraded quality. The precision and recall of the judge itself have to be measured separately.

Batch processing, and why the estimate and the invoice disagree

Work where you can give up latency can be sent down a separate asynchronous path at a much lower rate. Many providers apply roughly half price in exchange for a time allowance. Overnight batch classification, log summarization, embedding regeneration, evaluation set scoring — anything where no human is waiting is a candidate. Separating the interactive path from the batch path at the code level means you never have to re-decide later which workload belongs where.

Finally, the common reasons the calculator output does not match the actual invoice.

System prompts and tool definitions are input tokens on every request too. Attach twenty tools and their entire schema is billed every time. Simply pruning the tools you do not use can visibly reduce input tokens.

Retries and aborts are billed as well. If a user closes the window mid-stream, the tokens generated up to that point are still charged. A retry after a JSON parse failure means that request is paid for twice. A 5 percent parse failure rate is a 5 percent cost increase.

Estimating token counts from character counts goes badly wrong depending on the language. For sentences with the same meaning, Korean commonly produces more tokens than English. Counting with the real tokenizer, or logging the provider token count response field, is the only trustworthy method.

Multimodal inputs follow separate conversion rules. A single image becomes hundreds to thousands of tokens depending on resolution, so laying it on top of a text-based estimate throws the estimate off badly.

And your logs need to record, per request, input tokens, output tokens, cache read tokens, cache write tokens and the model name. Without those five fields there is no way to prove after the fact which optimization saved how much.

Closing — the five log fields come first

The most common failure in cost optimization is not pulling the wrong lever, it is not knowing the effect after you pull it. If you are not recording usage by token type at the request level, every improvement proposal ends at "it seems like it helped."

With that record in place, the order is largely settled. Check first whether output tokens dominate the invoice; if they do, start with the output format. Then split the prompt into fixed and variable parts and fix the ordering. Up to here the quality risk is close to zero. Routing and RAG are trades made against quality, so decide them with the accuracy loss written next to the saving in the same size.