Memory and State Management

Memory and State Management

Short-term Memory

Short term memory

10 of 69 in this book · 3 min read

What it is

State that is established during a session and discarded when the session ends — the user is flying to Berlin on Thursday, the file under edit is the one named in turn three, the last two calls to the pricing service failed. It is derived rather than transcribed: nobody said "my working file is X", the system concluded it. Its boundary is the session, so it has to survive dozens of turns and several tool calls while earning no right to exist tomorrow.

Why it is not working memory

Working memory is bounded by attention within a single step: the handful of items in front of the model right now, competing for one budget and rewritten every iteration. Short-term memory is bounded by the session: a pool of established facts, most of which are not in front of the model on most turns and are fetched into the active set only when a step needs them. The practical consequence is that they fail differently — working memory fails by crowding out, short-term memory fails by expiring, by being keyed wrongly, or by never being written at all.

Why it matters

Without a session tier, every fact a conversation establishes has only two possible homes. It can live in the prompt, where it competes for attention and dies at the next compaction, or it can be written to durable memory, where "I am testing against staging today" becomes a standing belief about the user. The session tier exists so that a temporary circumstance can be remembered reliably and forgotten automatically, which is the behaviour people actually expect from a conversation.

Key decisions

  • What constitutes a session — a connection, a thread, an idle timeout, an explicit user action. Every lifetime, isolation, and expiry property follows from this one definition, and it is the hardest to change later.
  • Whether session state is written explicitly by the agent through named operations or extracted automatically from the dialogue after each turn. Extraction catches things the agent would not have thought to record and invents things nobody said.
  • Where it lives: a process variable, which is fast and dies with the worker; or a store keyed by session, which survives a deploy and costs a round trip per access.
  • Whether anything graduates to durable memory at session end, what the promotion criterion is, and whether the user is shown what was kept.
  • What the agent does when session state is missing mid-conversation — re-derive silently, ask again, or fail — given that a user can tell the difference between all three.

Failure modes

  • Session state lives in process memory, a rolling deploy lands mid-conversation, and the agent re-asks everything it was told ten minutes earlier.
  • The session boundary is an idle timeout, so a user who returns from lunch finds an assistant that still shows the transcript but has forgotten the task, which reads as amnesia rather than as a new session.
  • State is keyed by user rather than by session, and the same person's two open tabs overwrite each other's working file.
  • Everything is promoted at session end, so the durable store fills with one-day circumstances that will be asserted as facts for a year.
  • Nothing is promoted ever, so a genuine correction the user made — "my name is spelled with one L" — has to be made again in every future session.
  • There is no session tier at all and the design relies on the model re-reading the whole transcript each turn, so cost grows with the square of the turn count and the established facts are re-derived, differently, every time.
  • Session state is written on every turn without a size bound, and by turn eighty it is larger than the conversation that produced it.

Related capabilities

Applies: Production readiness · Design questions