Conversation Memory — Patterns

Record the messages, derive the input

Problem. The thing sent to the model is a selection: truncated, summarised, re-templated, with retrieved material spliced in. If that assembled form is what gets stored, the archive becomes a snapshot of a prompt template rather than of a conversation, and it stops being readable the moment the template changes.

Approach. Keep the message sequence as the system of record and treat the model input as a derived artefact produced by a named, versioned assembly step. Store alongside each assistant turn the identifier of the assembly version and the list of message ids and retrieved items that went into it — not the rendered text. Replay is then a function of the record plus the assembler at a stated version, and the record remains legible to anything that understands messages.

Trade-off. Exact reproduction now depends on keeping old assembler versions runnable, which is a real maintenance burden and will eventually fail for inputs that no longer exist. For incidents where the literal bytes matter, teams end up storing the rendered form too, for a short window, and paying for both.

Branch on edit instead of overwriting

Problem. Editing a message and regenerating an answer are ordinary user actions, and the obvious implementation replaces the old content. The previous answer — the one the complaint is about, the one the user acted on — is then gone.

Approach. Model the thread as a tree of message nodes with parent pointers, where an edit creates a sibling rather than replacing a node, and the thread carries a pointer to the currently active leaf. Reading a conversation means walking from the active leaf to the root. Every historical path stays addressable, which makes "what did it say before I edited this" answerable without a separate audit system.

Trade-off. Every reader now needs the path-walk, and any code that assumed a flat list — export, search indexing, token counting, the mobile client — has to be changed. Storage grows with abandoned branches, and deciding when an abandoned branch may be collected is a policy question nobody wants to own.

Compaction with a preserved original

Problem. A thread that has run for hundreds of turns cannot be replayed into a prompt, so some middle portion has to be compressed. If compression is applied to the record, the product has destroyed its own evidence to save tokens at inference time.

Approach. Make compaction produce a new artefact attached to the thread — a summary node covering an explicit, closed range of message ids — and leave the covered messages in place. Assembly prefers the summary for that range; export, audit, and the user's own scrollback continue to read the originals. Because the summary names its range, it can be invalidated and regenerated when the messages in that range change, and a reader can always tell which part of a prompt was a summary.

Trade-off. The record keeps growing even though prompts stop growing, so storage cost is unbounded and must be addressed by retention rather than by compaction. Two representations of the same stretch of conversation exist, and any inconsistency between them is a bug that only shows up when someone compares them.

Thread identity assigned by the server

Problem. When clients mint thread identifiers or infer them from a session, the same conversation acquires different identities on different devices and after a reconnect, and there is no reliable key to attach memories, summaries, or deletion requests to.

Approach. Make thread creation an explicit server-side operation returning an opaque, stable identifier, and require clients to send it with every message along with a client-generated idempotency key per message. Reconnects and retries then resolve to the same thread and do not duplicate turns. Every derived artefact — summaries, extracted beliefs, audit entries — references that identifier, so "everything belonging to this conversation" is a computable set rather than a guess.

Trade-off. A round trip is required before the first message can be sent, which hurts perceived latency on the very first interaction, and offline-first clients need a local provisional identity that must later be reconciled. Idempotency keys only help if every client actually generates them, which means the oldest client version in the field sets the floor.