Why LLM Features Cost More Than They Should: Our Field Notes on Prompt Caching, Cache Breakpoints, and the Timestamp That Kills Every Hit

Key takeaways

• Prompt caching is a provider-side store of the processed prefix of a request, so a later request with an identical prefix skips re-processing it. With the Anthropic Messages API it is opt-in per content block: adding cache_control with type ephemeral to a block makes everything up to and including that block the cached prefix, and a request may carry at most four breakpoints.

• A cache read is billed at 0.1x the base input token price, a five-minute cache write at 1.25x, and a one-hour write at 2x. The extra 0.25x paid on a five-minute write is repaid by the first hit inside the window, because each hit saves 0.9x.

• The cache key is the exact serialized prefix in the fixed order tools, then system, then messages. A timestamp in the system prompt, a tool array built by iterating a Map, or a swapped model id all invalidate the cache from the point of change onward.

• Every Messages API response reports input_tokens, cache_creation_input_tokens, and cache_read_input_tokens. Cache hit rate equals read divided by the sum of read, creation, and fresh input, and it is the only honest way to know whether caching is working.

• Prompt caching does not reduce output token cost and does not fix a bloated prompt. Deleting four thousand tokens of context nobody reads beats caching them at 0.1x forever.

The feature was a support agent with a large tool schema and a long API reference pinned into the system prompt, which is exactly the shape prompt caching is built for. We added a breakpoint, deployed, and the cost per conversation did not move. The usage object was blunt about it: cache_creation_input_tokens was populated on every request and cache_read_input_tokens was zero on every request. We were paying 1.25x to write an entry that nothing ever read.

What is prompt caching, and what are we actually paying for?

Prompt caching lets the provider reuse the work it already did on a prefix of a prompt, so identical leading tokens are billed at a steep discount instead of being processed again. It splits the input bill into three separate line items rather than one: fresh uncached input, cache writes, and cache reads. Reading those three numbers is the whole discipline.

Two constraints decide whether a breakpoint does anything at all. The first is the minimum cacheable prefix, which is 1024 tokens on the larger Claude models and 2048 on the smallest ones; below the minimum the cache_control marker is ignored and no error is raised, which looks exactly like caching being broken. The second is the time to live: the default entry lives five minutes and every read refreshes that window, so steady traffic keeps a prefix warm indefinitely without paying to write it again. Passing a ttl of one hour extends the window at a higher write price.

Where does the cache breakpoint belong?

Put the breakpoint on the boundary between what never changes per request and what always changes. The provider serializes tools first, then the system prompt, then messages, so anything meant to be cached must sit before anything volatile. A large tool schema is usually the biggest static block in an agent request and is the best first thing to place behind a breakpoint.

In practice we mark the last static system block with cache_control set to ephemeral, keep the tool definitions above it, and leave the user turn below it. The cold call then reports non-zero cache_creation_input_tokens and every warm call reports non-zero cache_read_input_tokens.

For a multi-turn conversation we use two breakpoints instead of one. The first sits at the end of the tools and system region, which never changes for the life of the deployment. The second sits at the end of the last completed turn, so the growing transcript is cached incrementally and each new turn pays full price only for the tokens the user just added.

Why does a cache hit rate drop to zero?

A cache hit requires the prefix to be identical byte for byte, so anything that varies per request destroys every hit from that point onward. Our own failure was one interpolated clock reading in the system prompt, added months earlier so the agent could reason about business hours.

Interpolated volatile values are the most common cause: a timestamp, a request id, a user name, or a session id rendered into the system prompt changes the prefix on every call. Non-deterministic ordering is the second: a tool array built by iterating an object or a Set, or a body serialized with unstable key order, produces a different prefix from identical data.

Model changes start the cache cold, because entries are keyed per model and an alias that silently points somewhere new behaves like a different model. Middleware that rewrites the request breaks the prefix even when application code is unchanged. Trimming a conversation from the front to fit a context budget rewrites the start of the prefix and invalidates everything after it.

The fix is always the same shape: freeze the system prompt, and move volatile facts such as the current time into the user turn below the breakpoint.

When is a one-hour cache TTL worth 2x the write price?

The break-even point is pure arithmetic on the published multipliers, not a benchmark. A five-minute write costs 0.25x more than an uncached request and each hit saves 0.9x, so the first hit inside the window already pays for the write. A one-hour write costs 1x more, so it needs roughly two hits to come out ahead.

• No caching costs 1x input and fits prefixes below the model minimum or genuinely one-shot calls.

• A five-minute cache costs 1.25x to write and 0.1x to read, breaks even after one hit, and fits chat turns and agent loops where a live user keeps traffic flowing.

• A one-hour cache costs 2x to write and 0.1x to read, breaks even after two hits, and fits a long document reused across a work session or a batch spread over an hour.

Concurrency changes the math in one specific way. The entry is created by the request that writes it, so firing ten cold requests in parallel means paying the write price ten times. When our services fan out, they send one warm-up call first and start the fan-out only after it returns.

How do we measure cache hits instead of guessing?

We log the three input counters on every response and compute a hit rate, because caching fails silently and looks identical to caching that was never enabled. The ratio is emitted as a metric, and a drop right after a deploy is treated as expected rather than alarming, since editing a prompt is by definition a cache invalidation.

Through the Vercel AI SDK the same numbers arrive under provider metadata instead of a top-level usage object. Marking a message with providerOptions.anthropic.cacheControl sets the breakpoint, and providerMetadata.anthropic on the result carries cacheCreationInputTokens and cacheReadInputTokens. If those fields are absent, the request never reached the provider in a cacheable shape.

What does prompt caching not fix, and how do other providers handle it?

Caching is an input-side optimization only. It does not change model output, it does not reduce output token price, and it does not make a badly scoped prompt cheap. The single biggest cost win we found was not the cache at all; it was deleting a stale section of a reference document that no answer had ever cited.

• OpenAI applies prompt caching automatically to prompts above roughly 1024 tokens, with no explicit breakpoint to place, and reports cached tokens in the usage payload.

• Google Gemini exposes both implicit caching and an explicit context caching API, where a cached content handle is created with its own TTL and referenced by name.

• Anthropic is the explicit one: the caller chooses the breakpoints, which costs a design decision and buys precise control over what is cached.

The portable lesson survives all three: structure the request static-first, keep the volatile parts last, and verify with the provider's own token counters.

FAQ

Q: Does prompt caching change the model's output?

A: No. Prompt caching reuses the processed prefix and does not alter sampling, so the same prompt and parameters behave the same whether the prefix was read from cache or processed fresh.

Q: Can another organization read a cached prompt?

A: No. Cache entries are scoped to a single organization and keyed on the exact prefix, so there is no cross-organization sharing of cached content.

Q: Does the five-minute TTL reset on every hit?

A: Yes. Each cache read refreshes the five-minute window, so continuous traffic keeps a prefix alive without ever paying the write price again.

Q: Why do cache creation tokens appear on every request when nothing changed?

A: Something in the prefix is changing, most often an interpolated value or a non-deterministic key order. Hash the serialized prefix of two consecutive requests and diff the strings; the difference is always visible once you look at the bytes.

Q: Should tool definitions or the system prompt be cached first?

A: Tools, because they are serialized before the system prompt and are usually the largest static block in an agent request. Placing the breakpoint after the system prompt covers the tools as well, since the cached prefix is everything up to the breakpoint.