Semantic Caching — Patterns

Namespace the index by everything that changes the answer

Problem. Similarity is computed on query text, but two identical questions can have different correct answers depending on who asked, which documents were visible to them, which tools were available, and which model version replied.

Approach. Partition the index by a composite namespace — tenant, entitlement set, locale, model version, prompt version — and search only within the caller's partition. Treat it as a hard partition enforced by the store, not as a metadata field that a post-filter is expected to check.

Trade-off. Hit rate falls, sometimes sharply, because every partition starts cold and sparse. That is the correct direction to err: a cache miss costs a generation, and a cross-partition hit costs an incident report.

Threshold calibrated on labelled near-miss pairs

Problem. The similarity threshold is typically set to a round number found in an example, and then adjusted upward or downward based on hit rate — a metric that improves monotonically as the cache becomes more wrong.

Approach. Build a labelled set from real traffic: pairs of queries annotated for whether one's answer genuinely serves the other, deliberately oversampling adversarial pairs that differ by a negation, a date, a quantity, or an entity name. Sweep the threshold over that set and choose the operating point by acceptable false-hit rate, then report hit rate as the consequence rather than the target.

Trade-off. The labelled set is manual work and ages with the traffic. It also only measures the pairs you thought to include, so an unrepresented query family can sit on the wrong side of a threshold that looks well calibrated.

Cacheability declared at the call site

Problem. Inferring from the request text whether an answer is safe to reuse is unreliable: a personalised question can be phrased generically, and a request with side effects looks like any other.

Approach. Make cacheability an explicit property set by the code issuing the request, defaulting to not cacheable. Only call sites answering from shared, slow-changing knowledge opt in, and opting in requires naming the knowledge source so invalidation has something to hang on. Any request that reads user state, reads live data, or triggers a tool is excluded by construction.

Trade-off. Conservative defaults mean most traffic never benefits, and the opt-in list needs periodic review to be worth having. It is a smaller cache that is defensible, instead of a larger one nobody can reason about.

Invalidation by provenance tag

Problem. When a source document changes, every cached answer derived from it is stale, but nothing in the cache key mentions the document — the key is a query embedding.

Approach. Record, on each cache entry, the identifiers of the documents and tools that contributed to the answer. On a content update, delete by that tag. Pair it with a conservative time-to-live so that entries whose provenance was never recorded, or whose sources changed outside the tracked path, still expire on their own.

Trade-off. Provenance must be captured at generation time, which couples the cache to the retrieval path, and tag-based bulk deletion is an expensive operation on some stores. The time-to-live remains the real safety net, and setting it short enough to be safe erodes much of the hit rate the cache was built for.