Prompt Caching vs Semantic Caching: the Honest Tradeoffs

Prompt caching stores the model's already-processed prefix so that re-sending the same context costs a fraction of the original price. Semantic caching stores finished answers and reuses one when a new question looks similar enough to an old one. They get compared as if you pick one, and that framing causes most of the trouble: prompt caching is exact-match infrastructure that cannot return a wrong answer, and semantic caching is a similarity bet that can.

We build recal, a local-first assistant that runs retrieval and models on the user's own Mac, so cache behaviour is something we design around rather than read about. Nothing below is a pitch. The mechanics stand on their own.

Key takeaways

• Prompt caching matches the exact bytes of your prompt prefix. Any byte that changes anywhere in the prefix invalidates everything after it.
• Semantic caching matches a vector against a threshold. That threshold is doing a job cosine similarity was never built for, which is where wrong answers come from.
• The failure modes are asymmetric. A prompt cache miss costs you full-price input tokens. A semantic cache false hit costs you correctness.
• On Claude models, cache reads run about 0.1x the base input price, while writes cost 1.25x at the five minute TTL and 2x at the one hour TTL. Two requests break even on the short TTL, three on the long one.
• They compose cleanly, because they save different things. A semantic hit skips the call. Prompt caching makes the miss cheap.

What is prompt caching?

The provider keeps the processed form of a prefix of your request and reuses it on the next request that starts with the identical bytes. The render order is tools, then system, then messages, so a breakpoint on the last system block covers your tool definitions too.

The whole model follows from one property: it is a prefix match. The cache key derives from the exact bytes up to each breakpoint, so a single differing character at position N invalidates every breakpoint at or after N. That is why prompt caching is mostly a prompt-architecture problem rather than a configuration problem. Stable content has to physically precede volatile content.

One detail that catches people: the minimum cacheable prefix is model-dependent, and it is not monotonic across generations. On current Claude models it ranges from 512 tokens up to 4096. A 3000 token prompt caches on some models and silently does not cache on others, with no error, just a cache-write count of zero.

What is semantic caching?

You embed the incoming query, run a nearest-neighbour search over previously answered queries, and if the closest match scores above your similarity threshold you return its stored answer without calling the model at all.

That is a bigger prize than prompt caching offers. A prompt cache hit still runs a generation, it just charges less for the shared prefix. A semantic cache hit replaces the generation with a vector lookup. On repetitive traffic the savings are not comparable.

The cost is that you have taken on a judgment call. Somewhere there is a number, and above that number you are willing to hand a user an answer written for a different question. Every semantic cache is a bet that queries which sit close together in embedding space have the same correct answer. Usually true. When it is false, nothing in the system notices.

Why does semantic caching serve wrong answers?

Because the collision is on an axis that embeddings compress.

Consider "how do I cancel my subscription" against "how do I pause my subscription". Those are near-identical sentences that differ by one operator word. Cosine similarity encodes what a sentence is about, and both are about subscription state, so they score high. The one distinction that matters, which operator the user asked for, is exactly the distinction that barely moves the vector. Negation behaves the same way. So does swapping a comparative.

No threshold cleanly separates that pair, which is why tuning the threshold feels like whack-a-mole: raise it and you lose real hits, lower it and the collision comes back. It also explains a result that shows up when people actually measure this instead of nudging the number. In a public study of roughly 210,000 requests across three datasets, a perfect oracle verifier would have allowed noticeably more cache hits at the same error rate, while a generic off-the-shelf verifier landed near random on short queries. The gap was not a lack of verifier capacity. It was that the signal the verifier needed had already been flattened.

The fix that holds up is to stop letting confusable operators share a bucket. Extract the action and the object, key the cache on that, and run similarity only inside the bucket. Cancel cannot land on pause's key, because they are different keys.

That moves the trust to the extractor, which is a fair objection, and the answer is that its errors are not symmetric with the original ones. If the extractor mislabels a word, the query gets a different key and misses, and a miss costs one model call. If the threshold fails, a wrong answer ships. The extractor does not have to be right, it has to be consistent, because the same text keys the same way on write and on read and still finds its own entry. Keying on the surface verb rather than a lemma buys that consistency outright, at the cost of more misses on inflected forms.

What silently breaks prompt caching?

Almost always something in the prefix that varies per request. The usual suspects, in rough order of how often they turn up:

Pattern	Why the cache never hits
datetime.now() or a timestamp in the system prompt	Prefix bytes change on every request
A UUID or request ID placed early in the content	Same, every request is unique
json.dumps(d) without sort_keys=True, or iterating a set	Non-deterministic serialization
A session or user ID interpolated into the system prompt	Per-user prefix, nothing shared across users
Conditional system sections behind feature flags	Each flag combination is a separate prefix
A tool list that varies per user or per mode	Tools render first, so nothing downstream caches

Three subtler ones are worth knowing. Caches are model-scoped, so switching models mid-conversation starts from cold. In long agentic turns, a breakpoint only walks back a limited number of content blocks looking for a prior entry, so a turn that appends dozens of tool-use and tool-result pairs can quietly outrun the lookback window. And on a parallel fan-out, the entry is only readable once the first response has started streaming, so firing N identical requests at once means all N pay full price. Send one, wait for the first token, then release the rest.

Diagnosing all of this is one number. Read the cache-read token count off the response. If it stays at zero across requests you believe share a prefix, you have an invalidator, and diffing the rendered bytes of two requests will find it.

Prompt caching vs semantic caching, side by side

	Prompt caching	Semantic caching
What it matches	exact prefix bytes	vector similarity over a threshold
What it saves	re-processing the shared prefix	the entire model call
Can it be wrong	no	yes
Cost of a miss	full-price input tokens	one model call
Who operates it	the provider	you
What you tune	breakpoint placement, prompt ordering	threshold, embedding model, verifier
Direction of failure	closed	open
Honest catch	invisible invalidators make it silently useless	a false hit is indistinguishable from a real one

Which one should you actually use?

Turn prompt caching on whenever you have a stable shared prefix and repeat traffic inside the TTL. There is no correctness risk, the ceiling is a wasted write premium, and the work is prompt ordering rather than tuning. It should be the default, and for most teams the honest answer is that they have it enabled and are still losing most of the benefit to one timestamp.

Reach for semantic caching when the question space is narrow and closed, and when a wrong answer is cheap. A support bot over a fixed FAQ corpus is a good fit. Anything where the operator carries the meaning is not, and the high-stakes verbs are a short list you can write down: cancel, delete, refund, downgrade, transfer. Those recur across domains, so treating them as a closed class you refuse to bucket across is more practical than trying to solve the general case.

Can you use both at once?

Yes, and it is the sensible arrangement, because they save at different layers. Put the semantic cache in front, where a hit skips the model entirely, and let prompt caching sit behind it making every miss cheaper. One measurement caveat: if you add a semantic cache to a system that already had prompt caching working well, the reported savings will look better than the semantic layer earned. Compare against a prompt-cache-only baseline, not against no cache at all.

Does any of this change on-device?

The economics invert, which changes which lever is worth pulling. There is no per-token bill on a local model, so what you are caching is latency and battery rather than dollars, and the provider-side prefix cache becomes a KV cache you own and can keep warm yourself. That makes the safe lever cheaper than it is in the cloud.

The risky lever does not get any safer. A semantic cache serving the wrong answer costs exactly the same locally as it does behind an API, and you have lost the one thing local-first is supposed to buy, which is an answer you can trust because you can see where it came from. This is why recal keeps a warm local cache for retrieval and generation but does not reuse a stored answer for a merely similar question. The tradeoff is real and it goes the other way too: we pay for more generations than a semantic cache would.

FAQ

Is prompt caching just KV caching?

Closely related. A KV cache reuses attention state within a generation, and prompt caching is that idea persisted across requests and billed, with the constraint that the reuse only applies to an exact prefix.

Does a reranker fix semantic cache collisions?

It helps and it is the natural first reach, but it is domain-dependent in a way that is easy to miss. A cross-encoder scoring the stored query against the incoming one does well on longer conversational text and much less well on short keyword-dense queries, which is precisely where the operator-word collisions live.

Why is my cache-read token count zero?

Something in the prefix differs between requests. Check for a timestamp or ID in the system prompt first, then serialization order, then whether your prompt clears the model's minimum cacheable length.

Does prompt caching change the model's output?

No. It changes how the prefix is billed and processed, not what the model sees. A semantic cache does change the output, because it returns a previously generated one.

Related reading

• BM25 vs vector search for local RAG, on the same theme of lexical signal that embeddings flatten.
• Best local embedding models for RAG, if you are choosing the model whose similarity scores you are about to trust.
• Best local reranker models, for the verification layer discussed above.

Written with AI assistance and edited by a human. Pricing multipliers and cacheable-prefix minimums cited here are from Anthropic's prompt caching documentation and are current as of publication; check your provider's current numbers before budgeting against them.