Semantic Caching

What it is

Reusing a previous model response for a new request whose meaning is close enough, where closeness is measured by embedding similarity rather than byte equality. A new query is embedded, the nearest stored query is retrieved, and if the distance is under a threshold the stored answer is returned without calling a model. The threshold is the entire design: it is a tunable trade between hit rate and the chance of answering a question that was never asked.

Why it matters

Real traffic is repetitive in meaning and almost never repetitive in bytes. "How do I reset my password" and "i forgot my password how to reset" are the same support question and share no exact-match cache key. A semantic cache converts that repetition into saved cost and a response in milliseconds instead of seconds. Unlike an exact cache, it can also be confidently wrong, which is why it is a product decision rather than an optimisation.

Why it is unlike an ordinary cache

A conventional cache is either correct or empty: the key either matches or it does not. A semantic cache has a third state — a near miss returned as a hit. Everything that makes caching safe elsewhere, from key equality to deterministic invalidation, is weakened here, and the compensating controls have to be designed rather than inherited.

Key decisions

• What belongs in the cache key beyond the query text. Tenant, user entitlements, locale, persona, selected tools, model version, and the identity of any retrieved documents all change what a correct answer is. Omitting any of them creates cross-contamination.
• Where the similarity threshold sits, and how it was chosen — ideally from a labelled set of query pairs scored for whether one answer would genuinely serve the other.
• Which request classes are cacheable at all. Questions about shared, slow-changing knowledge are; anything that reads per-user state, current data, or produces a side effect is not.
• How entries expire and are invalidated when the underlying knowledge changes, since a semantic cache has no natural key to invalidate by.
• Whether a cached answer is disclosed as cached, and whether users can force a fresh answer.

Failure modes

• Two queries differing only by a negation, a date, a quantity, or a named entity embed close together, and the cache answers the opposite question with full confidence.
• Tenant identity is left out of the key and one customer receives an answer generated from another customer's context — a data-leak incident, not a cache bug.
• The threshold is tuned on hit rate alone, hit rate improves, and answer quality degrades with no metric watching it.
• Cached answers outlive the documents they were grounded in; the retrieval corpus is updated and the cache keeps serving the superseded answer indefinitely.
• Cache lookup itself costs an embedding call and a vector search, so on low-hit-rate traffic the cache adds latency and cost to every request while saving on few.
• Personalised or stateful answers are cached because the request text looked generic, and a later user sees a response containing another user's details.
• A poisoned or adversarial query is cached once and then served repeatedly, amplifying a single bad generation across many users.

Related capabilities

• Embedding and Reranking Models — the embedding model defines what "similar" means here
• Model Gateway — a cache placed there serves every caller, and sees every tenant's traffic
• Token Budget Management — a hit removes a call's spend entirely
• Model Routing — a hit short-circuits the routing decision before it is made
• Model Versioning — an entry is only valid for the model version that produced it

Applies: Production readiness · Design questions