Memory Deduplication

What it is

Detecting that two or more stored items say the same thing and collapsing them to one, with a defensible rule for which survives and what happens to everything that referenced the others. The duplicates arise mechanically: the same fact restated across three sessions, a write retried after a timeout, two agents independently recording the same event, an import run twice. They are one claim appearing several times, not several claims that need reconciling.

Why it is not consolidation

Deduplication is identity-preserving. Afterwards the store asserts exactly what it asserted before, in fewer rows, and every surviving statement is one that an input actually made. Consolidation is generative: its output states something none of the inputs stated, and it discards detail on purpose to get there. Letting a deduplication job merge "similar" items with a model is how it quietly becomes a summariser and starts losing facts it was only supposed to be counting.

Why it matters

Duplicates damage a memory store twice over. They crowd the retrieval budget, so one restated item displaces several distinct ones, and they read to a model as corroboration — a thing mentioned once in three sessions arrives looking like three independent confirmations. They also break correction and erasure: fixing or deleting a fact fixes or deletes one of its four copies, and the other three keep being retrieved.

Key decisions

• The identity rule: exact text, normalised text, a structural key, or embedding similarity above a threshold. Only the last catches paraphrase, and only the last merges things that were never the same.
• Whether the check runs at write time, costing an index lookup on every write, or as a periodic sweep, leaving duplicates retrievable until it runs.
• Which copy survives — earliest, latest, longest, highest confidence — and what that does to the timestamp and the provenance the rest of the system reasons about.
• Whether a merged item keeps a count or a list of contributing sources, since repetition is a genuine signal that naive merging destroys.
• How references from elsewhere — derived items, provenance chains, user-facing identifiers — to a removed copy are repaired rather than left dangling.
• Whether merging is reversible, given that a wrong merge is otherwise indistinguishable from the duplicate never having existed.

Failure modes

• The similarity threshold merges "the invoice was approved" with "the invoice was not approved", and no human ever sees that it happened.
• Near-duplicates differing in exactly the field that matters — a version number, an account identifier, a date — are collapsed into whichever arrived first.
• The surviving copy is the earliest, so a restated and updated fact loses the update and keeps the original wording.
• The repetition count is discarded, and a belief held strongly because it was confirmed many times becomes indistinguishable from an offhand remark.
• Merging keeps only the surviving copy's provenance, and the item's remaining source turns out to be the weakest of the ones that were merged.
• The sweep is partitioned for efficiency, and duplicates that span partitions are never found, so the store looks deduplicated and is not.
• A write-time check uses an approximate index that has not caught up with the last few seconds of writes, so rapid restatement produces exactly the duplicates the check was meant to prevent.
• Deduplication runs across scope boundaries and merges one person's item into another's because the text was identical.

Related capabilities

• Memory Consolidation — the generative operation this must not become
• Memory Conflict Resolution — what to do when the items disagree rather than repeat
• Memory Retrieval — the budget duplicates consume and the corroboration they fake
• Memory Provenance — the source lists a merge has to preserve
• Memory Deletion — why unmerged copies make erasure incomplete

Applies: Production readiness · Design questions