Prompt Caching Explained: Cut Cost and Latency (Anthropic, OpenAI & Gemini)

Prompt caching is the single highest-leverage cost lever most teams never turn on. If your app resends a big system prompt, a tool schema, or a long document across many calls, you are paying full price to recompute tokens the provider already processed seconds ago. Caching stores the computed state of that prefix so repeat calls skip the work: you pay a steep discount on the cached tokens and get a faster first token. The catch is that the three major providers implement it three different ways, with different control models and different pricing shapes, and one careless token near the front of your prompt can quietly bust the whole thing.

What prompt caching actually does

Every time you call an LLM, the model has to process your entire prompt from the first token to compute the internal state it needs to generate a response. If your prompt starts with the same 8,000-token system prompt on every single call, the model redoes that same work every time, and you pay for it every time. Prompt caching breaks that cycle. The provider stores the computed state of a prompt prefix so that a later call beginning with the exact same tokens can reuse it instead of recomputing.

The key word is prefix. Caching does not fingerprint your whole prompt and look for an exact match; it matches from the front and reuses as far as the tokens stay identical. That is why the reusable, stable parts of your prompt, such as the system prompt, the tool schema, a long reference document, or a block of few-shot examples, are exactly the parts worth caching, and why they belong at the very start of the request. The model still sees every token, so the output is unchanged. What changes is that the cached tokens are billed at a steep discount and the first token comes back faster because the expensive prefill work is skipped. This concept is identical across Anthropic, OpenAI, and Google; only the controls and pricing differ.

If you are approaching this from a cost angle, caching sits alongside the other big levers we cover in how to cut LLM API costs, and it is the single biggest one for agentic and RAG workloads.

Three providers, three control models

The mental model splits cleanly along one axis: how much control you have, and how much you are asked to do.

Anthropic is explicit and opt-in. Nothing is cached unless you ask. You add a cache_control marker, called a breakpoint, and everything up to and including that block is cached. Anthropic walks the prompt in a fixed order, tools then system then messages, up to your marked block, and you can set up to four breakpoints per prompt. This is more work, but it gives you precise control over exactly which segments get cached and lets you cache stable-but-not-first content deliberately.

OpenAI is automatic. There is no API change and no opt-in. Any prompt of 1,024 tokens or more on a supported model is eligible, and the platform matches the longest previously-computed prefix, starting at 1,024 tokens and growing in 128-token increments. You get the benefit for free structurally; your only job is to structure your prompt so the stable prefix is long and identical across calls.

Gemini offers both. Implicit caching is on by default with no setup and gives a large discount on repeated prefixes. Explicit caching lets you create a CachedContent object and reference it by handle, which guarantees a hit but adds a per-token-hour storage fee for the time-to-live you set. That storage fee is the twist that makes Gemini explicit caching behave differently from the other two, as we will see in the break-even math.

Prices and multipliers above are pricing shapes as of July 2026; exact per-token dollar figures move fast across model tiers, so always confirm against each provider's live pricing page. For Anthropic specifically, we keep a running breakdown in Claude API pricing.

The pricing shapes that matter

Forget the dollar figures for a moment and learn the shapes, because the shapes are what determine your strategy.

Anthropic: pay a premium to write, then reads are almost free. Writing a cache entry costs more than a normal input token, roughly 1.25x base input for the standard 5-minute TTL and about 2x for the optional 1-hour TTL. Once written, reading that cache costs only about 0.1x, a tenth of the base input price. The default 5-minute window is a sliding one: every time the cached prefix is read, the clock resets, so an active session keeps the cache warm indefinitely. If you use both TTLs, longer-lived entries must appear before shorter-lived ones in the prompt.

OpenAI: no write premium, discounted reads. Historically OpenAI charged nothing extra to populate the cache, and cached input is billed at a discount versus standard input. OpenAI documents cached input at roughly 50 percent off standard input for most models; the exact fraction is model-dependent, so treat it as a fraction of normal input and check the model you actually use.

Gemini: implicit is free to populate, explicit charges rent. Implicit caching bills you normally to create the cache and gives about a 75 percent discount on cache hits with no separate storage cost. Explicit caching offers a comparable discount but charges a storage fee per token-hour for the TTL you hold, with a default of one hour. That rent is the reason explicit caching is not automatically a win.

The break-even rule

Here is the part most articles skip: when does caching actually save money, and when does it quietly cost you? The answer follows directly from the pricing shape, and it is different for each provider.

Take Anthropic's multipliers. A cache write costs about 1.25x base input; each read costs about 0.1x. Without caching, N calls that each resend the same prefix cost N times the base input for that prefix. With caching, you pay 1.25x once to write, then 0.1x for each of the remaining reads. Set the two equal and the crossover lands almost immediately: by the second reuse of the prefix within the TTL, caching is already net-positive, and every read after that is nearly free. That is why caching is a no-brainer for any workload that touches the same prefix more than once in a five-minute window. This is illustrative math derived from the published multipliers, not a provider-quoted figure, but the direction is robust.

OpenAI is even simpler: with no write premium, you are ahead on the very first cache hit, the second time the prefix is seen. Gemini implicit behaves the same way. Gemini explicit is the outlier. Because you pay rent per token-hour regardless of how many times you read, a large cache that gets reused only a handful of times before the TTL expires can cost more than paying full price each call. Explicit caching wins only when reuse density is high enough that the discount on many reads outweighs the storage rent.

This break-even logic is why caching pairs so well with agent and coding workloads specifically, where the same instructions and context get resent on every turn. We go deeper on that pattern in cut token costs for coding agents.

Cache-busting anti-patterns

The prefix-matching rule cuts both ways. Because the cache matches from the front and stops at the first differing token, anything that changes near the start of your prompt invalidates everything downstream, even if 99 percent of the content is identical. These are the silent hit-rate killers, and they are almost always accidental.

• A timestamp or UUID in the system prompt. Injecting the current date and time, a request ID, or a session UUID at the top of the prompt changes the first block on every call, so nothing ever caches. Move dynamic values to the end of the prompt, or omit them if the model does not need them.
• Non-deterministic tool ordering. If your tool schemas are serialized from a hash map or a set, their order can shuffle between calls. Since Anthropic caches tools first, a reordered tool array busts the entire cache. Serialize tools in a stable, sorted order.
• Per-user data before shared context. Placing user profile fields, account IDs, or personalization ahead of the shared system prompt means every user, and often every request, has a unique prefix. Put the shared, static block first and the per-user block last.
• Rotating few-shot examples. Randomly sampling or shuffling examples each call defeats caching for the whole example block. Freeze a canonical example set, or place the variable examples after the stable ones.
• Traffic below the warm-cache threshold. Even a perfectly structured prompt never benefits if calls arrive further apart than the TTL. Infrequent traffic pays the write cost repeatedly and never gets a warm read. Batch related requests, or accept that low-volume endpoints will not cache.

The single design rule that prevents almost all of these: stable content first, variable content last. Structure every prompt so the ordering is fixed system prompt, then tools, then long static context, then finally the dynamic user turn. Latency benefits ride on the same structure, since a warm prefix is exactly what skips the expensive prefill step; providers report up to roughly 80-85 percent faster time-to-first-token on cache hits, which is a big part of the story in how to reduce LLM latency.

Caching across models on one gateway

Here is the operational problem. Anthropic wants explicit cache_control breakpoints. OpenAI wants you to do nothing but structure the prompt well. Gemini wants either implicit reliance or a managed CachedContent object. An application that calls providers directly has to implement all three, and re-implement them every time it adds a model. That is exactly the kind of per-provider glue an LLM gateway exists to absorb.

A gateway can normalize the mechanism per model: insert breakpoints for Anthropic, pass prompts through to OpenAI's automatic caching untouched, and manage cache objects for Gemini, so the same application code gets caching benefits consistently no matter which model the request routes to. That is the DataLLM Lab positioning: one OpenAI-compatible endpoint at https://www.datallmlab.com/v1, 300+ models on a single key, so you write your prompt structure once and let the gateway apply the right caching path per provider. Whether you route a coding agent to Qwen3 Coder Next, high-volume classification to DeepSeek V4 Flash, or reasoning work to Claude Opus 4.8, the caching contract stays the same from your side.

FAQ