How Prompt Caching Works in Claude and OpenAI

When two requests start with the same text -a long system prompt, tool definitions, a user prompt— the provider can reuse the work it already did on that shared prefix instead of reading it again. Reusing that work is a cache hit — it reduces the bill for those matched tokens by up to ~90% for repetitive tasks (according to both Anthropic and OpenAI). The discount stops the moment the text diverges: everything after the first difference, plus the model’s output, is billed in full. So the practical question is how much of each request you can keep identical from one call to the next. This page shows what qualifies, and how to structure prompts to hit the cache more often — on both Claude and OpenAI.

What token caching actually is

Caching is an exact prefix match over the rendered prompt. The provider assembles your request into model input, then reuses the leading portion it has processed before — up to a breakpoint you mark (Claude) or the longest matching prefix it finds (OpenAI). If that rendered prefix is identical, it’s read from cache instead of re-processed; a change anywhere inside it invalidates the hit from that point onward. What matters is the input the model sees, not the literal HTTP bytes, keep the rendered prefix stable.

It is called prompt caching (or token caching) because what gets cached is the model’s processed representation of the input tokens not the output. You still pay full price to generate the response; you save on reading the repeated input.

What you cache and why it helps:

Real prompts are mostly stable: a long system prompt, a fixed tool catalog, retrieved documents, few-shot examples — followed by a small, changing user question. Without caching, that entire preamble is re-tokenized and re-processed on every call. With caching, the stable preamble is read from cache and only the new tail is processed at full price.

Caching with Claude: Render order & Breakpoints

The API assembles the cacheable prompt prefix in this order, before cache lookup:

Claude supports two prompt-caching styles: automatic caching and explicit cache breakpoints. With automatic caching, you add a top-level cache_control field and Claude places the breakpoint at the last cacheable block. With explicit caching, you place cache_control: {"type": "ephemeral"} on a specific content block. In both cases, Claude caches the full prefix rendered before and including the breakpoint. That prefix is assembled in Anthropic’s documented render order — tools, then system, then messages — not simply the raw order of fields in your HTTP request. So a breakpoint on a later system or message block can include earlier tools and system content as part of the reusable prefix.

Explicit Cache Breakpoint

Automatic top-level caching is available on the Claude API, Claude Platform on AWS, and Microsoft Foundry. On Bedrock and Google Cloud, use explicit cache breakpoints.

Walk through: a cold write, then a cache hit

Read the receipt. A healthy cached route should show large cache_read_input_tokens. In simple fixed-prefix reuse, cache_creation_input_tokens may drop to 0 after the first call. In growing conversations, you may see both cache reads and cache creation as Claude reads the old prefix and writes the newly extended prefix.

Explicit Breakpoints — How many breakpoints, and where

You can place cache_control more than once — up to 4 markers per request. A marker isn't stored server-side — you re-declare it in the fresh JSON you send each turn. The classic layout uses two: one pinned on the system prompt, one riding the newest message.

Multi-turn: the prefix grows, and each turn reads the whole history

Because the API keeps no hidden conversation memory, you resend the conversation history on each turn. Prompt caching makes this cheaper when the earlier history remains identical. With automatic caching, Claude moves the cache point forward as the conversation grows: the next request can read the previously cached conversation prefix, then process and write the newly extended tail. With explicit caching, you can achieve the same pattern by moving or adding breakpoints carefully, while staying mindful of the 20-block lookback window.

So the cheap “read” part keeps growing while the full-price part stays tiny — a long chat costs less per turn than you’d expect, not more. Notice the read column:

Watch the 20-block lookback. Anthropic’s prompt-caching docs note a breakpoint only searches back ~20 content blocks to find a prior cache entry.

Silent invalidators

Anything non-deterministic in the prefix quietly kills every hit:

• datetime.now() / Date.now() in the system prompt
• a uuid4() or request ID placed early
• non-deterministic serialization of anything that lands in the prompt — tool schemas, embedded JSON examples, retrieved documents, tool results (e.g. json.dumps() without sort_keys=True, or iterating a set)
• a per-user ID interpolated into the system prompt
• conditional system sections (if flag: system += …)
• tools that vary in content or order between requests

Golden layout: frozen content first (system prompt, deterministically-serialized tools), volatile content last (timestamps, per-request IDs, the actual question). Inject dynamic context as a later message, never into the system-prompt header. Don’t count on raw-JSON byte identity — make the rendered input deterministic: stable tool order, stable schema text, stable message and document order.

Cost & Limits

TTL is a sliding expiry with default 5 minutes (a 1-hour option exists for bursty traffic), and every read refreshes the clock. Break-even, ignoring output tokens and any uncached tail: the 5-minute cache beats uncached input on the 2nd use of the prefix; the 1-hour cache generally needs two later reads — three uses total. A hit shows up as a large cache_read_input_tokens in the response's usage block (the ledgers above).

Minimum cacheable prefix

The prefix must clear a minimum size that varies by model — below it the marker is ignored and nothing caches

Prompt caching on OpenAI

The prefix-match idea is common. OpenAI does not use a cache_control breakpoint — prompt caching is automatic on supported models, with hit reporting via cached_tokens and optional routing/retention controls.

How it works

• Automatic — no Claude-style cache marker is required.
• Eligibility starts at 1,024 prompt tokens; the cache grows in ~128-token increments.
• Cache hits require an exact prefix match over the rendered input.
• Static content (instructions, tools, few-shot, shared context) first; volatile content (user data, timestamps, request IDs, fresh retrievals) later.
• Hits reported via usage.prompt_tokens_details.cached_tokens.
• prompt_cache_key improves routing for many requests sharing a long prefix; where supported, prompt_cache_retention configures retention behavior.

What’s different from Claude

• No explicit breakpoint placement — you influence the cacheable region only by ordering content.
• No separate caller-visible write premium. The first (uncached) call is billed at the normal input rate.
• Provider-managed retention — in-memory and extended policies depending on model/API support (prompt_cache_retention where available); no per-request 5m/1h write-tier choice like Claude's.
• Model-dependent read discount. Current OpenAI docs describe input-token cost reductions up to ~90%; older models were ~50% — verify the pricing page for your model.

The two knobs OpenAI provides

OpenAI won’t let you place the cache boundary, but it does expose two parameters that decide where a cached prefix lives and how long it survives. Both matter because the cache is per-server, not global — a prefix is only a hit if your request lands on a machine that still holds it.

prompt_cache_key — routing affinity

prompt_cache_key is an optional routing-affinity hint. OpenAI routes requests using a hash of the initial prompt prefix, and the key is combined with that hash. When many requests share the same long prefix, using a stable key such as a tenant, prompt-template, or agent identifier can improve the chance that related requests land on a machine with the relevant cached prefix. It is not a manual cache key or a guarantee of a hit; it influences routing. Keep each unique prefix/key combination below roughly 15 requests per minute, otherwise requests may overflow to additional machines and cache effectiveness can drop.

prompt_cache_retention — how long it lives

Chooses the eviction policy for the cached prefix.

• **in_memory (the short tier) — a prefix stays warm for roughly 5–10 minutes of inactivity, up to a 1-hour** ceiling, then it's evicted.
• **24h (extended retention) — keeps the prefix warm up to 24 hours**.
• Why it helps. For a big prefix reused on a slow cadence — a system prompt hit a few times an hour, an overnight batch, a low-traffic tenant — in_memory evicts between uses so every call is cold. 24h covers those gaps.

Practical guidelines for developers

What to log, which KPIs to compute, and how to add the cache block — for Claude and OpenAI.

Effectiveness is roughly (cached-prefix size) × (reuse frequency within TTL): a big stable preamble reused often is a large win; a small or rarely-repeated prefix is marginal, and on Claude possibly net-negative because of the write premium.

Worth caching

• Chatbots with a long fixed system prompt.
• Agents re-sending a growing conversation each turn.
• RAG where many questions share the same retrieved context.
• Batch jobs over a shared instruction block.

Skip it

• One-shot requests whose first ~1K tokens differ every time.
• Nothing to reuse — on Claude, a cache_control marker just pays the write premium for zero reads.
• Prefixes below the model’s minimum cacheable size (they silently never cache).

1. What to log per request

2. Helpful KPIs to compute

3. How to add the cache block

Claude:

Automatic Caching

Explicit Caching

OpenAI:

References

Anthropic (Claude)

• Prompt caching guide — breakpoints, render order, TTLs, read/write multipliers, limits.
• Pricing — per-model input/output and cache rates.
• Models overview — model IDs and context windows.
• Prompt Caching Cookbook

OpenAI

• Prompt caching guide — automatic caching, cached_tokens, prompt_cache_key / prompt_cache_retention.
• API pricing .
• Prompt Caching in the API .

Note: Some of the details above were generated with the help of Claude Code. Provider pricing, limits, and parameters change frequently — please verify the specifics against the official docs under references before relying on them.