Model Management and Inference
Semantic Caching — Stack
Semantic caching › Stack
Technology landscape as of 2026-10-10. Knowledge cutoff May 2026.
Selection criteria
- Lookup latency at p99. The cache must be dramatically faster than generation on a miss as well as a hit, because every request pays the lookup. Budget single-digit milliseconds, not tens.
- Metadata filtering before similarity. The store must restrict candidates by tenant, locale, model version, and entitlement before computing nearest neighbours. A system that filters after retrieval will eventually return a neighbour it should never have seen.
- Threshold transparency. Whether the raw similarity score is exposed to the application, so the decision stays yours and can be logged, rather than hidden behind a boolean hit.
- Invalidation primitives. Tag-based or prefix-based bulk deletion, plus per-entry time-to-live, since semantic entries have no natural invalidation key.
- Distance metric and normalisation. Cosine on normalised vectors behaves differently from inner product on unnormalised ones; a threshold is meaningless without knowing which.
- Write amplification. Whether every miss writes an entry, and what bounds the index's growth and its memory footprint.
- Eviction policy. Whether it is least-recently-used, time-based, or absent — an index that only grows becomes the latency problem it was meant to solve.
Options
| Option | Best for | Watch out for |
|---|---|---|
| Redis with vector search | Teams already running Redis that want lookup and time-to-live in one system | Index lives in memory; size it deliberately and confirm filter-then-search ordering |
| Dedicated vector databases (Qdrant, Weaviate, Milvus, pgvector on PostgreSQL) | Reusing the retrieval store you already operate, with rich metadata filtering | Tuned for recall over large corpora, not for a few-millisecond point lookup; measure before assuming |
| GPTCache | A ready-made semantic cache with pluggable embedding, storage, and evaluation stages | Adds its own similarity-evaluation layer with its own defaults; read what the default actually is before trusting it |
| Gateway-level semantic cache features (LLM gateway products) | Getting a cache in front of every service without touching application code | The gateway chose your key, threshold, and embedding model — verify all three carry tenant isolation |
| Provider prompt caching | Reusing a long stable prefix at reduced cost with zero correctness risk | Not semantic at all: it is exact-prefix reuse, it still calls the model, and it complements rather than replaces this capability |
| Exact-match cache on a normalised query string | High-volume identical queries, as a safe first layer with no false-hit risk | Hit rate collapses on natural phrasing variation, which is the entire reason this capability exists |
A sound deployment usually layers these: exact-match first, semantic second, generation last, with the layer that answered recorded on the response.
Models
The embedding model is a load-bearing choice here and a different one from retrieval. The job is to separate queries whose answers differ, so what matters is sensitivity to negation, numbers, dates, and named entities — precisely the signals that general-purpose retrieval embeddings are often trained to blur. Prefer a small, fast model, because lookup latency is paid on every request, and evaluate candidates on a labelled set of near-miss pairs drawn from your own traffic rather than on retrieval benchmarks. Treat the embedding model as part of the cache key: changing it invalidates every stored entry, because the geometry the threshold was calibrated against no longer exists.