Skip to content
Published on

How to Actually Cut Your LLM API Bill — Why "90% Off From Caching" Is 25% on the Invoice

🇰🇷KO🇺🇸EN🇯🇵JA🇨🇳ZH
Split View필사 모드
Share
Authors

Introduction — A Discount Rate Is Not an Invoice Cut

"Turn on prompt caching and you save 90%" is a staple of LLM cost articles. It is half true. It is a fact that a cache read costs one-tenth of the base input price — but that is the discount on a single line item, the cache read. For the whole invoice to drop by that much, every dollar you spend would have to be cacheable input tokens. In reality it is not.

Let me give the answer first. A discount reduces the total only in proportion to that line item's share of the invoice. In the worked example Anthropic itself publishes in its pricing docs, a one-hour session on Claude Opus 4.8 using 50,000 input tokens and 15,000 output tokens costs $0.705. If 40,000 of those 50,000 input tokens turn into cache reads, the total becomes $0.525. The input cost is cut 72%, from $0.25 to $0.07 — but the total drops only 25.5%. That is because output tokens were already eating most of the cost.

So this post does not list discount rates. It computes the invoice instead. Why caching works in the first place (the KV-cache story inside the transformer) is already covered in How LLM Caching Works, and how the serving engine actually runs it is covered in Prefill/Decode Disaggregation and Goodput. Here we look strictly at the economics of buying the API from the outside.

About the Numbers in This Post — Prices Change Often

LLM cost articles rot fast because prices change often. So let me nail down the ground rules first.

  • Every price in this post was pulled directly from each provider's official pricing page on July 17, 2026. Not a single number comes from a secondary source, a summary, or memory.
  • Prices change without notice. This post even contains the evidence — Claude Sonnet 5's introductory price is valid only through August 31, 2026, and rises on September 1. OpenAI's cache-write charge is new in the GPT-5.6 generation (earlier models were free). Both are cases where "a sentence that used to be correct" is now wrong.
  • I cannot guarantee these numbers are still correct at the moment you read this. Before making any decision that affects your bill, open each provider's price sheet yourself. The links are in the references at the bottom.

What does not change is not the prices but the structure of the arithmetic. That structure is what this post is really trying to teach.

Where Your Invoice Comes From — Output Tokens Dominate

Before deciding what to optimize, you have to see where the money is leaking. An LLM API invoice almost always breaks into three line items — uncached input, cache reads, and output.

Here is the fact most people miss. The output token price is several times the input price. These numbers come from Anthropic's price sheet as of July 17, 2026.

Model                     Input (per 1M)   Output (per 1M)   Output/Input
Claude Opus 4.8           $5               $25               5x
Claude Sonnet 5 (intro)   $2               $10               5x
Claude Haiku 4.5          $1               $5                5x
Claude Fable 5            $10              $50               5x

The Claude family is a consistent 5x. On the OpenAI side the multiple is larger — on the price sheet as of July 17, 2026, gpt-5.6-sol is $5 input / $30 output at short context, a 6x ratio, and gpt-5.6-luna is $1 / $6, again 6x.

Why does this matter? Take the worked example from Anthropic's docs verbatim (Opus 4.8, 50,000 input and 15,000 output tokens).

Line item   Tokens    Share    Cost      Cost share
Input       50,000    76.9%    $0.25     40%
Output      15,000    23.1%    $0.375    60%
Total       65,000             $0.625

Output is only 23.1% of the tokens but 60% of the cost. If you look at the token counts and conclude "input is the problem," you have it exactly backwards.

So the first practical rule is not optimization but measurement. The Anthropic API returns input_tokens, output_tokens, cache_creation_input_tokens, and cache_read_input_tokens in the response usage, and OpenAI returns cached_tokens (cache reads) and, from the GPT-5.6 generation on, cache_write_tokens (cache writes). Until you multiply each of these four numbers by its unit price to get a per-line-item amount, every optimization is a guess.

How Much Does Prompt Caching Actually Save?

Now let us finish the calculation from the opening. The Managed Agents section of Anthropic's pricing docs has a table placing before-and-after caching side by side. Below I transcribe that document's numbers and check the arithmetic.

[No caching] Opus 4.8, 50K input / 15K output, 1-hour session
  Input            50,000 x $5  / 1M = $0.25
  Output           15,000 x $25 / 1M = $0.375
  Session runtime  1.0 hr x $0.08     = $0.08
  Total                               = $0.705

[With caching] 40K of the 50K input is cache reads
  Uncached input  10,000 x $5 / 1M        = $0.05
  Cache reads     40,000 x $5 x 0.1 / 1M  = $0.02
  Output          15,000 x $25 / 1M       = $0.375
  Session runtime  1.0 hr x $0.08         = $0.08
  Total                                   = $0.525

Savings      = 0.705 - 0.525 = $0.18
Savings rate = 0.18 / 0.705 = 25.5%

The cache-read price is plainly 10% of the base input (i.e. "90% off"). And yet the total dropped only 25.5%.

Three things are worth pointing out.

First, the $0.08 session runtime is a Managed Agents-only line item. The plain Messages API does not have this row. So if you isolate the token cost alone, it goes from $0.625 to $0.445, a 28.8% cut. Either way, it is nowhere near 90%.

Second, this example assumes the cache is already warm. There is no cache-write cost in the calculation. On the first request you pay the write charge on top, so the actual first call is more expensive. We compute that break-even in the very next section.

Third — and this is the real conclusion — after caching, output's share of the token cost climbs to 0.375 / 0.445 = 84.3%. By erasing the input line item, caching actually increases output's dominance. The moment you turn caching on properly, the next lever is always output.

How Many Reads Until Caching Pays Off?

Caching is not free because of the write charge. As of July 17, 2026, the multipliers Anthropic's docs state, relative to the base input price, are these.

Cache operation      Multiplier         Lifetime
5-min cache write    1.25x base input   5 min
1-hour cache write   2x base input      1 hour
Cache read (hit)     0.1x base input    same as the preceding write

Say you send the same prefix N times. Without caching you pay N (a relative value setting the base input price to 1); with caching you pay 1 write + (N-1) reads. Working it out.

5-min cache (write 1.25x)
  N=1: no cache 1.00  vs  cache 1.25          -> cache loses
  N=2: no cache 2.00  vs  1.25 + 0.10 = 1.35  -> cache wins
1-hour cache (write 2x)
  N=2: no cache 2.00  vs  2.00 + 0.10 = 2.10  -> still a loss
  N=3: no cache 3.00  vs  2.00 + 0.20 = 2.20  -> cache wins

In other words, the 5-minute cache clears break-even from the first read, and the 1-hour cache from the second read. This matches exactly the break-even Anthropic's docs state, and it checks out directly with the arithmetic above.

The practically important point is this. If your traffic does not bring the same prefix back at least twice within 5 minutes, caching increases your cost. This is exactly the case of turning caching on for a low-frequency endpoint that gets a few hits a day and then asking "why isn't this getting cheaper?" That said, Anthropic states that for the 5-minute cache the lifetime is refreshed at no extra cost each time the cache is used, so as long as traffic keeps flowing the 5 minutes keep extending.

On the OpenAI side the rules differ by generation. Per OpenAI's docs, models before GPT-5.6 have no extra charge for cache writes, and from the GPT-5.6 generation on a cache write costs 1.25x the uncached input price. The old received wisdom that "OpenAI cache writes are free" breaks in the latest generation. A cache read is $0.5 against gpt-5.6-sol's $5 input — exactly 10%. As a result, in the latest generation the two companies' cache pricing structures (write 1.25x / read 0.1x) have become essentially the same.

When Caching Doesn't Kick In at All

The most deflating failure is turning caching on and having nothing happen at all. If the prompt falls short of the minimum length, the cache simply does not engage.

Per Anthropic's docs (July 17, 2026), the minimum cacheable prompt length in the Claude API varies by model.

Claude Fable 5, Mythos 5                       512 tokens
Claude Mythos Preview, Opus 4.7              2,048 tokens
Claude Opus 4.6, Opus 4.5                    4,096 tokens
Claude Haiku 4.5                             4,096 tokens
Claude Opus 4.8, Sonnet 5, Sonnet 4.6,
  Sonnet 4.5                                 1,024 tokens

The docs attach a crucial caveat here — a prompt below the minimum length is processed without caching even if you attach cache_control, and no error is raised. Meaning it fails silently. To check whether the cache engaged, look at cache_creation_input_tokens and cache_read_input_tokens in the response; if both are 0, it did not.

One more caution. This minimum can differ by platform. Anthropic's docs separately note that on Amazon Bedrock the minimum length for Fable 5 and Mythos 5 is 1,024 tokens. For Bedrock's per-model minimums, failure behavior, and field names, you have to consult the AWS docs.

OpenAI is simpler in structure — per the docs, caching applies automatically to prompts of 1,024 tokens or more, and a cache hit occurs only on an exact prefix match. Requests under 1,024 tokens come back with cached_tokens at 0.

From this comes a design rule both companies emphasize. Put the fixed content (system prompt, examples, tool definitions) first, and the changing content (user input) last. Because the cache engages only on a prefix match, the moment you put a single timestamp line at the very front, everything after it falls out of the cache. Images and tool definitions must be identical across requests too.

Is the Batch API's 50% Real?

It is. And all three providers share the same number. Here is what I confirmed in each company's docs as of July 17, 2026.

  • Anthropic: the Message Batches API bills "all usage at 50% of the standard API price." Both input and output.
  • OpenAI: the Batch API is "a 50% cost discount" versus the synchronous API. On the price sheet this checks out for gpt-5.6-sol at short context — input $5 → $2.5, output $30 → $15.
  • Google: the Vertex AI pricing page states "Gemini models get a 50% discount in batch mode," and this checks out for Gemini 3.1 Pro Preview — input $2 → $1, output $12 → $6.

Unlike caching, the batch discount applies to both input and output. So on an output-dominated invoice, batch cuts the total far more than caching does. Run the earlier Opus 4.8 example (token cost $0.625) through batch and it becomes $0.3125, exactly half. Compare that with caching's 28.8%.

Of course it is not free. The price is time and certainty.

Anthropic: the docs say "most batches complete within 1 hour," but also that results become accessible "when all messages are complete or after 24 hours, whichever comes first," and specify that a batch expires if processing does not finish within 24 hours. An important caveat is attached — processing can slow depending on demand and request volume, in which case more requests may expire after 24 hours. Expired requests, however, are not billed.

OpenAI: the completion window (completion_window) can currently be set only to 24h, and the batch statuses include expired ("did not complete within the 24-hour window").

In short, batch is not "50% off" but a package of "50% off + 24-hour-expiry risk." You cannot use it for a request a user is waiting on. Nightly report generation, bulk document classification, embedding re-indexing, running an eval set — for work like this it is almost always a win.

For Workloads That Can't Use Batch — Flex

There is a lesser-known option here. OpenAI's Flex processing is a synchronous call at batch prices. In OpenAI's own words, Flex tokens are "billed at the Batch API rate," with the prompt-caching discount applied on top. On the price sheet, too, gpt-5.6-sol's Flex rate (input $2.5 / output $15) is exactly identical to the Batch rate.

Since you just set service_tier to flex instead of building a 24-hour batch-file workflow, it fits non-interactive work that is awkward to move to batch.

The cost is clearly documented. Flex is cheap in exchange for accepting "slower response times and occasional resource unavailability," and it is currently in beta with limited model support. When resources are unavailable you get back 429 Resource Unavailable, and the docs specify that in this case you are not billed. The SDK's default timeout is 10 minutes, so for long jobs you have to raise the timeout, and OpenAI offers two strategies on failure — retry with exponential backoff, or switch service_tier to auto and retry with standard processing.

One-line summary: for non-interactive work that still cannot wait 24 hours, Flex fills exactly that gap.

Does Switching to a Cheaper Model Save That Much? — The Tokenizer Trap

The logic of model routing is simple. Easy requests to a cheap model, only hard requests to an expensive one. The price ladder as of July 17, 2026 is this.

Anthropic (per 1M tokens, input / output)
  Claude Fable 5              $10 / $50
  Claude Opus 4.8              $5 / $25
  Claude Sonnet 5 (intro)      $2 / $10   <- through 2026-08-31
  Claude Sonnet 5 (from 9/1)   $3 / $15
  Claude Haiku 4.5             $1 /  $5

OpenAI (per 1M tokens, short context, input / output)
  gpt-5.6-sol                  $5 / $30
  gpt-5.6-terra              $2.5 / $15
  gpt-5.6-luna                 $1 /  $6

Haiku 4.5 ($1/$5) and Opus 4.8 ($5/$25) are exactly a 5x gap. Push 80% of your easy requests down to Haiku and you get a correspondingly large saving — the reason routing genuinely earns its keep.

But here is the trap. Per-token prices are comparable only when the tokenizer is the same. Anthropic's pricing docs carry this footnote — Opus models from Claude Opus 4.7 on, plus Fable 5, Mythos 5, Mythos Preview, and Sonnet 5, use a new tokenizer, and this tokenizer produces about 30% more tokens for the same text. (The docs add that "the exact increase depends on the content and the shape of the workload." Sonnet 4.6 and earlier models use the previous tokenizer.)

Let us compute how this flips a price comparison. Take the docs' "about 30%" at face value as 1.3x, and suppose you process the same text T.

Sonnet 4.6 ($3, old tokenizer) vs Sonnet 5 intro ($2, new tokenizer)
  Naive per-unit compare: (3-2)/3          = 33.3% cheaper
  Same-text basis:        3T vs 2 x 1.3T = 2.6T
                          (3-2.6)/3        = 13.3% cheaper   <- actual

If Sonnet 5 rises to $3 on September 1
  Same-text basis:        3T vs 3 x 1.3T = 3.9T
                          -> 30% more expensive than Sonnet 4.6

Opus 4.1 ($15, old) vs Opus 4.8 ($5, new)
  Naive per-unit compare: (15-5)/15        = 66.7% cheaper
  Same-text basis:        15T vs 5 x 1.3T = 6.5T
                          (15-6.5)/15      = 56.7% cheaper  <- actual

Some honest caveats belong here. This calculation is arithmetic resting on the assumption that the input text is identical, not a benchmark. Anthropic's docs explain that the new tokenizer "contributes to improved performance across a broad range of tasks," so you should not read the extra tokens as pure loss. A better model may finish the same job with a shorter output, or need fewer retries. And "about 30%" is most likely an English-based approximation — for a language with different tokenization characteristics, like Korean, there is no published figure for what the actual percentage is. There is no way around feeding your real prompts into each model's token-counting API and counting them yourself.

The rule that falls out is this. When switching models, compare not the rate card but "my real prompt's token count x unit price." Comparing unit prices alone across models from different generations is like subtracting two numbers in different units.

The Long-Context Surcharge — What Nobody Budgets For

Everyone knows that a long context costs more because there are more tokens. Far less known is that with some providers the unit price itself goes up. This is where the three companies split. Each figure comes from that company's own price sheet (as of July 17, 2026).

OpenAI's price sheet lists gpt-5.6-family rates in two sets, "Short context" and "Long context."

gpt-5.6-sol    short context   $5 / $30
               long context   $10 / $45     -> input 2x, output 1.5x
gpt-5.6-terra  short context  $2.5 / $15
               long context    $5 / $22.5   -> input 2x, output 1.5x
gpt-5.6-luna   short context   $1 / $6
               long context    $2 / $9      -> input 2x, output 1.5x

All three models are exactly input 2x, output 1.5x. But the price-sheet page does not say from how many tokens "long context" begins for the gpt-5.6 family. The same page marks the boundary on older-generation model rows, as in gpt-5.5 (<272K context length), but there is no such marking for the gpt-5.6 family. So I will not invent a threshold here — check it yourself.

Google's Vertex AI pricing page does state the threshold. Gemini 3.1 Pro Preview is $2 input / $12 output, but past 200K input tokens it becomes $4 / $18 (again input 2x, output 1.5x). And a footnote on this page has a sentence that can hurt — once the query input context exceeds 200K tokens, all tokens, input and output included, are billed at the long-context rate. Meaning if you cross the threshold by a single token, the unit price for that entire request changes.

Anthropic goes the other way. The pricing docs state that Fable 5, Mythos 5, Mythos Preview, Opus 4.8, Opus 4.7, Opus 4.6, Sonnet 5, and Sonnet 4.6 include the full 1M-token context at the standard price, and nail it down in parentheses — "a 900K-token request is billed at the same per-token rate as a 9K-token request." Prompt caching and the batch discount also apply at the standard rate across the entire context range.

This is not a benchmark comparison but a comparison of pricing structures. There is no common benchmark pitting the three companies' performance against each other here — only the billing rules each pulled from its own price sheet, set side by side. The practical implication is clear. On a workload with jagged context lengths, like RAG, requests that cross the threshold can quietly wreck your budget. With a provider that has a threshold, monitoring your context-length distribution — and trimming to just under the threshold — can by itself produce a large saving.

The Biggest Lever Is Output Tokens

The calculations so far kept pointing at the same place. Output is priced 5-6x higher and, once caching is on, climbs to 84% of the token cost. Yet output gets handled in a single paragraph at the very end of most cost articles. The order is backwards.

The ways to cut output tokens are almost boringly simple.

  • Set max_tokens to what you actually need. It is a ceiling, not a target, but it is the last line of defense against a runaway.
  • Force the output format. If a classification task needs a single label but the model is attaching three paragraphs of explanation, all of that explanation is billed at the 5x rate. Structured output (a JSON schema) is a cost tool before it is an accuracy tool.
  • Be aware of the price of "explain step by step." A prompt that draws out a long chain of reasoning spends output tokens to match. In an automation pipeline that needs only the final answer, it is simply waste. (For reference, Google's Vertex price sheet labels the output line item outright as "Text output (response and reasoning)" — meaning reasoning tokens are billed as output too.)
  • Count your retries. A failed call still costs you the output tokens it generated. A pipeline that retries 3 times on a parse failure has 4x the output cost.

There is one free saving on the input side too. Anthropic's pricing docs publish, per model, the token count of the system prompt that is automatically added when you use tools — for Opus 4.8 it is 290 tokens with auto/none, and for Opus 4.7 it is 675 tokens. On top of that come the tokens of the tool definitions themselves (e.g. the bash tool is 325 tokens on Opus 4.7 and 4.8). If you are shipping unused tools on every request, that is a tax you pay every time. Remember too that tool definitions sit near the front of the prefix, so caching can cover them.

So Where Do You Start?

There is an order. Work it from the top.

Step 1 — Calculate first. Scrape a recent day's usage fields and break out the amounts per line item (uncached input / cache reads / cache writes / output). If output is 60% or more, caching tuning is not the priority.

Step 2 — Find traffic you can move to Batch or Flex. If some requests are not being waited on in real time, 50% applies to both input and output. As a single lever, it is the biggest. If you can wait 24 hours, Batch; if not, Flex (on OpenAI).

Step 3 — Tighten output. Force the format, max_tokens, remove unnecessary reasoning, reduce retries. It is a 5-6x-priced line item, so the effect multiplies straight through.

Step 4 — Then caching. Turn it on only for traffic where the same prefix is reused 2 or more times within 5 minutes. Fixed assets first, changing data last. Confirm you cleared the minimum token count with the usage fields (it fails silently).

Step 5 — Routing. Push easy requests down to a cheaper model, but always compare on "my prompt's real token count x unit price." Across generations, comparing rate cards is meaningless.

Step 6 — Look at your context-length distribution. If you use a provider with a threshold, a long-context surcharge may be eating into your budget.

When You Should Not Do Any of This

This is the part to be honest about. Most of this post may not apply to you.

  • If your bill is a few tens of dollars a month, do nothing. The engineering time to design cache breakpoints and build batch pipelines costs far more than the savings. LLM cost optimization is something you start when the bill reaches the same order of magnitude as an engineer's salary.
  • If traffic is low-frequency, caching is a loss. The break-even calculation above is your answer, unchanged.
  • If quality is not nailed down yet, do not route. Push down to a cheap model, accuracy drops, retries and human intervention rise, and total cost goes up instead. Routing without an eval set is not cost optimization but a gamble on quality.
  • Do not use batch on a path a user is waiting on. It sounds obvious, but I do see attempts to move an interactive endpoint to batch, mesmerized by that "50%" number.
  • When you stack caching and batch, they multiply. Anthropic's docs state that the caching multipliers "stack" (apply multiplicatively) with other price modifiers, including the batch discount. That is good news, but it makes estimating costs confusing — you do not simply add 90% and 50% to get 140% off. Conversely, Fast mode cannot be used together with batch, and Managed Agents sessions do not get the batch discount, Fast mode, or the data-residency multiplier.

Closing

Reduced to one sentence: a discount rate reduces the total only in proportion to that line item's share of the invoice.

That a cache read is 10% of the input price is true. And that it cuts the total by only 25.5% — in Anthropic's own worked example — is also true. The two are not a contradiction. Output tokens were simply already eating 60% of the cost. Once you understand this structure, a headline like "90% savings" no longer moves you. Such a headline uses a line item's discount rate as if it were the invoice's reduction rate.

So the order is always the same — break the invoice into line items, touch the biggest item first, and explicitly compute the price of each lever (24-hour expiry, resource unavailability, the cache-write charge, quality degradation). And, including the numbers in this post, every price is as of today (July 17, 2026) and changes often. Start by opening the links below and checking for yourself. What does not change is not the prices but the arithmetic.

References

Discuss on Twitter • View on GitHub