Memory Retrieval

What it is

Choosing which of an agent's own stored memories to place in front of the model for the request in hand. The population being searched is the agent's own store — its recorded occasions, its standing beliefs about this person, its learned routines, its session state. That population is small, written by the system itself, heterogeneous in type and age, and dominated by two signals that have nothing to do with meaning: who the memory belongs to, and when it was formed.

Why it is not document retrieval

Searching a corpus answers "what does the organisation know about this topic" over a large, externally-authored, comparatively static body of text, where semantic similarity is the primary signal. Retrieving memory answers "what do I know about this person, this thread, this task" over a small, self-authored store that changes every session, where the scope key is a hard constraint and recency is often a better predictor than similarity. The same index can serve both, and the ranking policies have almost nothing in common.

Why it matters

Everything else in a memory system is latent until retrieval surfaces it: a store with perfect contents and poor retrieval behaves exactly like an empty store, only more expensively. It is also where memory's errors become visible to users, because an irrelevant memory placed in a prompt is worse than no memory at all — the model will assume it was included for a reason and will use it.

Key decisions

• Whether retrieval runs on every turn or only when the model asks for it. Always-on is predictable and is paid for on every request; on-demand is cheap and depends on the model recognising that it should ask.
• Whether the scope key is applied as a constraint on the query or as a feature in the ranking. Only the first is a security property; the second fails open under similarity pressure.
• How recency, frequency, and similarity are combined, and whether those weights were ever measured against outcomes or simply chosen once.
• The budget: how many memories, within what token allowance, and what happens when the single best candidate is larger than the allowance.
• Whether retrieved items are labelled to the model as memories with their age and origin, or pasted in as if they were part of the current conversation.
• Whether a retrieval that found nothing is distinguishable downstream from a retrieval that was never attempted, since the agent should behave differently in each case.
• What the query is built from — the last message, the whole turn, the active goal, or a model-written query — given that a one-word reply contains almost no signal.

Failure modes

• Scope is a ranking feature rather than a filter, and a sufficiently similar memory belonging to someone else outranks the correct one.
• A fixed top-k with no score floor returns five memories for every request whether or not any are relevant, and the model dutifully incorporates them.
• Recency dominates, so a correction made months ago is never retrieved while the superseded original keeps surfacing.
• Memories are injected unlabelled, and a year-old preference is treated as a current instruction with no way for the model to hedge.
• The query is the raw last user message, so "yes" retrieves nothing useful at precisely the moment the agent most needs context.
• The store is re-embedded with a new model and the query path still uses the old one, so scores become meaningless without any error being raised.
• Retrieval latency is added to every turn and nobody measures it separately from model latency, so a degraded index shows up as "the assistant got slower".
• Retrieved items are not deduplicated, so three copies of one fact fill the budget and read to the model as three independent confirmations.

Related capabilities

• Memory Access Control — why the scope key must be a constraint, not a signal
• Semantic Memory — one of the populations being ranked, with its own freshness rules
• Episodic Memory — the time-dominated population, where recency means something different
• Memory Deduplication — what stops near-identical items from consuming the budget
• Working Memory — where a retrieved item lands, and the budget it has to fit in

Applies: Production readiness · Design questions