Memory and State Management
Memory Retrieval — Stack
Memory retrieval › Stack
Technology landscape as of 2026-10-10. Knowledge cutoff May 2026.
Selection criteria
You are choosing a retrieval path over a small, heavily-filtered, constantly-changing store where correctness of the filter matters more than the sophistication of the ranking.
- Pre-filter versus post-filter. Whether scope and recency constraints are applied before candidate selection or after. Post-filtering silently reduces the result set and is not an access control.
- Freshness of the index. How long after a write an item becomes retrievable, since a memory formed this turn may be needed next turn.
- Delete visibility. How quickly a removed item stops appearing in results, which is both a correctness and a compliance property.
- Mixed-signal ranking. Whether the mechanism can combine similarity with explicit numeric signals such as age and use count without being bolted together in application code.
- Latency at small scale. Memory stores are small; a system engineered for billions of vectors may be slower at ten thousand than a simple index would be.
- Explainability. Whether the system can say why a specific item was returned and another was not, because this is how memory-driven quality incidents are diagnosed.
- Re-embedding cost. What it takes to rebuild the whole index when the embedding model changes, and whether serving can continue during it.
Options
| Option | Best for | Watch out for |
|---|---|---|
| Scope-keyed lookup with no ranking at all | Small per-person stores where everything relevant fits in the budget | Stops working the moment a store outgrows the budget, usually without anyone noticing the cutoff |
| Relational store with a vector index (PostgreSQL with pgvector, pgvectorscale) | Most memory systems: hard filters in WHERE, similarity in ORDER BY, one transaction boundary |
Index parameter tuning affects recall in ways that are invisible until measured; large indexes contend with transactional load |
| Dedicated vector database with payload filtering (Qdrant, Weaviate, Milvus, Pinecone) | Larger stores where filtered similarity is the dominant query | Filter semantics differ: some filter before search, some after, and the difference is a correctness issue, not performance |
| Lexical index alongside the vector index (BM25 in OpenSearch, Elasticsearch, or Postgres full-text) | Memories containing identifiers, names, and codes that embeddings handle poorly | Two indexes to keep consistent with the store, and two deletion paths that can disagree |
| Hybrid retrieval with rank fusion (reciprocal rank fusion over lexical and vector results) | Getting robustness across query types without tuning a single scoring function | Fusion hides which signal produced a result; debugging a bad retrieval becomes harder |
| Graph traversal from a resolved entity (Neo4j, FalkorDB) | Stores where relevance follows relationships rather than text similarity | Requires reliable entity resolution upstream; traversal depth is a tuning parameter with cliff-edge behaviour |
| Model-written retrieval queries issued as a tool call | Agents that know what they are looking for better than a template does | An extra model call per retrieval; the agent may not call it when it should, and the query is now non-deterministic |
Models
Model choice is in scope here, because embedding the store and embedding the query is a single joint decision and getting it wrong is silent. Treat the embedding model, its revision, any instruction prefix, the pooling strategy, and the normalisation as one inseparable identity: an index is valid only for the exact combination that produced it, and a change to any part of it invalidates every stored vector. Evaluate candidates on your own memory items rather than on general text benchmarks, because memory items are short, fragmentary, first-person, and full of names and identifiers — a profile that general benchmarks do not represent. Build the judged set from real retrievals where the correct memory is known, and measure recall at the depth you intend to actually use rather than at a depth you never retrieve to. Expect a modest open-weight model to be adequate for most memory stores, and expect the gap to a stronger model to appear on short, context-poor items rather than on long ones. Where a reranking model is used over memory candidates, verify that it improves ordering within the scope-filtered set rather than measuring it against an unfiltered population, which is the wrong task. Keep the model identity on every stored item so that a mid-migration store can serve both identities without mixing them.