Memory Consolidation — Patterns

Derived items name their input range

Problem. A consolidated item that floats free of its inputs cannot be invalidated when they change, cannot be regenerated, and cannot be checked. Over time the derived layer becomes an independent set of claims with no relationship to the record.

Approach. Every output names the exact set it covers — identifiers, or a closed time range plus a subject, plus the version of the prompt that produced it. When an input in that set is corrected, deleted, or superseded, the derived item is marked stale by construction rather than by anyone noticing. Regeneration is then a defined operation on a named set, and the share of retrievals served by stale derived items becomes a metric.

Trade-off. Maintaining coverage sets is bookkeeping on every pass, and the sets grow large for long-running subjects. Marking derived items stale on every input change also produces churn: a single corrected memory can invalidate several generations of derived content, and regenerating all of it may cost more than the correction was worth.

Bounded generation depth with forced re-grounding

Problem. When a pass may take derived items as inputs, each generation is a lossy rewrite of the previous one. Nothing stops the chain, and after several rounds the stored claims are confident, fluent, and unrelated to anything that was ever observed.

Approach. Record a generation number on every item and cap it. A pass may consolidate primary items freely; consolidating derived items is allowed only up to the cap, after which the pass must re-read the primary material within the coverage range. Where the primary material has been expired or deleted, the item cannot be re-grounded and is demoted rather than rewritten again.

Trade-off. Re-grounding means repeatedly reprocessing material that was already summarised, which is the cost consolidation existed to avoid, and it is highest for exactly the long-lived subjects where consolidation matters most. The cap is also arbitrary, and demotion of items that cannot be re-grounded deletes knowledge that may well have been correct.

Separate what was said from what is now claimed

Problem. One pass that both compresses and infers produces output in which "the user mentioned a deadline on Tuesday" and "the user works to tight deadlines" are the same kind of object, written with the same confidence, into the same store.

Approach. Split the pass into two explicitly different operations with different outputs and different rules. Compression produces a faithful reduction that asserts nothing its inputs did not, retains the subject's own framing, and can run unattended. Inference produces candidate claims that enter the belief store through the same admission path as any other belief, carrying their evidence and their uncertainty. Only the second needs review, confidence, and a contestation path, so the expensive controls apply only where they are warranted.

Trade-off. Two passes cost more than one over the same material, and the boundary is not crisp — any faithful summary involves selection, which is already a weak inference. Teams also tend to collapse the two back together under cost pressure, which reintroduces the problem without anyone recording that the decision was reversed.

Stage and swap rather than mutate

Problem. A pass that edits the live store as it goes leaves the store in an intermediate state visible to retrieval: half the occasions consolidated, the derived item not yet written, a failure in the middle leaving both halves inconsistent.

Approach. Write the pass's output to a staging area, validate it as a whole — generation depth, coverage completeness, scope compatibility of the merged inputs, absence of content that should not have been carried forward — and then make a single atomic transition that publishes the derived item and marks the inputs consolidated. A failure before the transition leaves the live store exactly as it was, and the staged output can be inspected to see what the pass would have done.

Trade-off. Atomicity across a derived item and a potentially large input set is easy in one database and hard once the vector index, the document store, and the provenance records are separate systems. Staging also doubles the storage for the duration of the pass and introduces its own cleanup obligation for passes that are abandoned.