Memory Retrieval — Patterns

Scope as a query constraint, never a score

Problem. When ownership is one feature among many in a ranking function, there exists some similarity score at which a memory belonging to another person outranks the correct one. The system is then one unusual query away from a cross-account leak, and nothing in the ranking will report it.

Approach. Split the pipeline into a filter stage and a ranking stage with a hard boundary between them. The filter stage applies ownership, tenancy, and any other access predicate as constraints that the retrieval engine evaluates before candidate selection, and it is the same predicate regardless of query. Only items that survive it are scored. Make the filter a single shared component so there is exactly one place where scope is expressed, and assert in tests that a retrieval with an empty scope returns nothing rather than everything.

Trade-off. Pre-filtering can degrade recall and latency in approximate indexes, because a highly selective filter forces the search to traverse further to find enough candidates, and some engines handle this far better than others. The hard boundary also removes the ability to rank across scopes in the cases where that is legitimately wanted, such as an administrator view, which then needs its own explicitly separate path.

Score floor before top-k

Problem. A fixed k returns k results by construction. On a request where the store holds nothing relevant, it returns the k least irrelevant items, and the model treats their presence as evidence that they matter.

Approach. Apply an absolute relevance threshold before truncating to k, and allow the result to be empty. Calibrate the threshold on labelled examples where the right answer is known to be "nothing", not only on examples where something should be found. Pass the emptiness downstream explicitly so that prompt assembly can omit the memory block entirely rather than rendering an empty one, which models read as a meaningful absence.

Trade-off. Similarity scores are not comparable across embedding models, across query types, or sometimes across index parameter changes, so the threshold is a per-configuration constant that will silently become wrong after an upgrade. An aggressive floor also drops marginal items that would occasionally have helped, and the loss is invisible because nothing records what was not returned.

Blend recency with relevance deliberately

Problem. Memories have an age and a usage history, and both matter differently per type: a stated preference from last year is probably still true, while a recorded system state from last year is almost certainly not. A single similarity ranking ignores this; a single recency ranking destroys it in the other direction.

Approach. Score candidates with an explicit combination of similarity, an age term whose decay rate is set per memory type, and a usage term, with the weights written down as configuration rather than embedded in code. Beliefs decay slowly or not at all, session-derived items decay quickly, recorded occasions are ranked by proximity to a relevant time rather than by absolute recency. Emit the component scores with the retrieval record so a surprising result can be attributed to a term.

Trade-off. Per-type decay rates are guesses until there is outcome data to fit them against, and outcome data for memory retrieval is hard to collect because the effect on the final answer is indirect. More terms also means more ways for a configuration change to shift behaviour across the whole product at once, with no per-request signal that anything changed.

Label every retrieved item in the prompt

Problem. Memories spliced into a prompt as plain text are indistinguishable from things the user just said. The model cannot weigh a stale item differently, cannot hedge, and cannot tell the user where a claim came from, because none of that information reached it.

Approach. Render each retrieved item with its type, when it was formed, how it was formed, and a stable identifier, inside a clearly delimited block that the system prompt explains. Give the model an explicit instruction about how to treat the block: that items in it are recollections that may be outdated, that it should prefer what the user says now, and that it may cite the identifier when asked why it believes something. The identifier also makes "that's wrong, forget it" actionable, because the user's complaint can be mapped to a row.

Trade-off. Labels cost tokens on every retrieved item and make prompts harder to read and harder to keep within budget. Models also comply with the handling instruction unevenly, so the label can produce false confidence that staleness is being accounted for when in practice it sometimes is not.