Prompt Caching for Agent Apps: A Practical Guide to Lower Cost and Latency

• Introduction
• Why prompt caching matters for agent apps
• How OpenAI and Anthropic differ
  • OpenAI
  • Anthropic
  • Quick comparison
• Prompt structuring patterns that work
  • Pattern 1: Put static assets first
  • Pattern 2: Standardize prompts across the team
  • Pattern 3: Push volatile context to the end
  • Pattern 4: Treat Anthropic breakpoints as reusable asset boundaries
• Simple implementation examples
  • OpenAI-style example
  • Anthropic-style example
• ROI scenarios
• Common mistakes
  • 1. Mixing variables into the front of the prompt
  • 2. Treating similar prompts as good enough
  • 3. Failing to separate stable prompt assets from experimental ones
  • 4. Putting Anthropic cache breakpoints too late
  • 5. Ignoring prompt_cache_key strategy on OpenAI
• Migration checklist
• Final takeaway
• References

Introduction

Agent applications reuse more prompt structure than most teams realize. System instructions, tool descriptions, policy text, shared examples, and workflow rules often stay the same across many requests. Usually only the final user task, document slice, or session state changes.

That makes prompt caching more than a small optimization. In agent products, it can be a structural advantage. If the repeated prefix of a prompt does not need to be processed from scratch every time, teams can reduce both latency and cost while keeping the same agent behavior.

This article explains how to think about prompt caching for real agent systems, using only official OpenAI and Anthropic documentation as factual grounding.

Why prompt caching matters for agent apps

Prompt caching becomes especially valuable when an application has all three of these traits:

1. Long system instructions
2. Repeated tool and policy definitions
3. Many requests that share the same prompt prefix

That is a very common pattern in support agents, internal copilots, workflow assistants, and document-heavy operational agents.

A typical agent request might contain:

• role and behavior rules
• safety and approval policy
• available tools and their descriptions
• shared examples
• the actual user request

Most of the expensive prompt tokens live in the repeated setup, not in the final user-specific line. If you can keep that repeated setup stable, caching becomes a direct lever for both product economics and response speed.

How OpenAI and Anthropic differ

Both vendors position prompt caching as a way to reduce time and cost for repetitive work, but they expose different design ideas.

OpenAI

OpenAI documentation says prompt caching is enabled for all recent models, gpt-4o and newer. The same documentation also says cache hits are only possible for exact prefix matches. In practice, that means static instructions and examples should go at the beginning of the prompt, while variable content should go at the end.

OpenAI also states:

• caching is enabled automatically for prompts that are 1024 tokens or longer
• prompt_cache_key can improve routing and cache hit rates

So for OpenAI, the main operational question is simple: can your team keep the shared prefix truly stable?

Anthropic

Anthropic documentation says prompt caching reduces processing time and costs for repetitive tasks. Anthropic supports both automatic caching and explicit cache breakpoints. The default cache lifetime is 5 minutes, and Anthropic also offers a 1-hour cache duration at additional cost.

Anthropic also documents the cache scope clearly: prompt caching covers tools, system, and messages up to the cache breakpoint.

So for Anthropic, teams usually think in terms of reusable prompt assets and where the cache breakpoint should sit.

Quick comparison

Prompt structuring patterns that work

Prompt caching rewards structure, not just prompt quality. The goal is to make the reusable prefix long, stable, and shared across many requests.

Pattern 1: Put static assets first

Move these items to the front whenever possible:

• system instructions
• shared policy
• tool descriptions
• common examples
• stable output rules

Move these items later in the prompt:

• user request
• per-session state
• request-specific variables
• changing document excerpts

[static system instructions]
[shared policy]
[tool descriptions]
[shared examples]
[variable user input]

Pattern 2: Standardize prompts across the team

Caching depends on identical prefixes, not nearly identical prefixes. If every squad rewrites the same support agent instructions in a slightly different way, cache efficiency drops quickly.

Good habits include:

• one template per agent family
• fixed tool ordering
• stable example count
• consistent versioning for shared prompts

Pattern 3: Push volatile context to the end

Retrieved passages, user profile fields, and fresh event logs often change on every request. If those move into the front of the prompt, the reusable prefix becomes smaller. Keep shared operating rules first, then attach variable context later.

Pattern 4: Treat Anthropic breakpoints as reusable asset boundaries

With Anthropic, it helps to think in terms of prompt assets that should be reused together. Long policy text, shared workflow docs, or large tool instructions are natural places to design around a cache breakpoint.

Simple implementation examples

OpenAI-style example

{
  "model": "gpt-4o",
  "prompt_cache_key": "support-agent-v1",
  "messages": [
    {
      "role": "system",
      "content": "You are the support operations agent. Follow the policy, use the available tools carefully, and produce a concise action summary."
    },
    {
      "role": "system",
      "content": "Policy: verify account scope, avoid irreversible actions without approval, and log the final decision."
    },
    {
      "role": "system",
      "content": "Tool guide: ticket_lookup, refund_policy_check, escalation_create."
    },
    {
      "role": "user",
      "content": "Customer asks whether order 48291 can be refunded after partial shipment."
    }
  ]
}

The first three blocks are the reusable prefix. If that shared prefix stays exactly the same, repeated requests have a better chance of benefiting from caching.

Anthropic-style example

{
  "model": "claude-sonnet",
  "system": [
    {
      "type": "text",
      "text": "You are the finance operations agent. Apply policy strictly and explain risk before action."
    },
    {
      "type": "text",
      "text": "Shared policy and workflow documentation goes here.",
      "cache_control": {
        "type": "ephemeral"
      }
    }
  ],
  "messages": [
    {
      "role": "user",
      "content": "Review this reimbursement request and tell me whether it should be approved."
    }
  ]
}

Here the shared policy block is treated as a reusable boundary. Different user questions can vary while the repeated setup remains stable.

ROI scenarios

Prompt caching does not pay off equally everywhere. The best results tend to show up in these cases:

• agents with long instructions
• workflows with many tool descriptions
• internal copilots that reuse the same operating prompt many times
• B2B products where many requests share a common setup
• document-heavy operations where the same background context is reused

The payoff is usually weaker when:

• prompts change significantly from the very beginning
• requests are consistently short
• prompt experiments constantly rewrite the shared prefix
• personalization data appears before the stable instructions

A simple rule works well: the longer and more stable the shared prefix is, and the more often it repeats, the stronger the business case becomes.

Common mistakes

1. Mixing variables into the front of the prompt

If ticket IDs, user names, or fresh logs appear before the shared instructions, the prefix becomes less reusable.

2. Treating similar prompts as good enough

Caching is sensitive to exact prefix reuse. Small wording edits, tool order changes, or example changes can reduce hit rates.

3. Failing to separate stable prompt assets from experimental ones

Teams often run experiments by changing the full system prompt. A better approach is to keep policy and tool definitions stable while isolating the part being tested.

4. Putting Anthropic cache breakpoints too late

If volatile user context appears before the reusable boundary, the cacheable portion loses a lot of value.

5. Ignoring prompt_cache_key strategy on OpenAI

OpenAI documents that prompt_cache_key can improve routing and cache hit rates. If a team uses it consistently by agent family or prompt version, operations tend to become easier to reason about.

Migration checklist

Use this checklist when moving an existing agent workflow toward prompt caching:

1. Split the current prompt into stable assets and variable assets.
2. Move shared instructions, tools, and examples into the prefix.
3. Push user-specific and request-specific content toward the end.
4. Standardize one prompt template per agent type.
5. For OpenAI, verify that a repeated prefix of 1024 tokens or more actually exists.
6. For OpenAI, define a consistent prompt_cache_key strategy by team, agent, or version.
7. For Anthropic, decide where reusable assets should end and where the cache breakpoint should sit.
8. Decide whether the default 5-minute reuse window is enough or whether the 1-hour option is justified.
9. After rollout, monitor cache behavior together with latency and per-request cost.
10. Keep prompt experiments separate from the stable cached prefix whenever possible.

Final takeaway

Prompt caching is not just a billing feature. For agent applications, it becomes a design discipline. OpenAI pushes teams toward exact shared prefixes and stable prompt ordering. Anthropic pushes teams to think carefully about reusable prompt assets and cache breakpoints.

If your agent product uses long instructions, repeated policies, and shared tool definitions, the first optimization step may not be a new model. It may be a better prompt layout.

References

• OpenAI Prompt Caching
• Anthropic Prompt Caching