Memory Deduplication — Patterns

Tiered identity: exact, then structural, then approximate

Problem. A single similarity threshold has to serve both a retried write that produced byte-identical text and a paraphrase recorded two months apart. Set it tight and the paraphrases remain; set it loose and distinct facts are destroyed.

Approach. Run the cheapest and safest test first and stop when it decides. A normalised hash catches literal repeats with no false positives and no model cost. A structural key catches the same typed claim written with different wording. Only what survives both goes to an approximate comparison, and that stage is tuned conservatively because it is the only one that can be wrong in the destructive direction. Record which tier made each merge decision, so precision can be measured per tier rather than for the system as a whole.

Trade-off. Three mechanisms is three things to maintain and three places a bug can hide, and the tiers disagree in edge cases in ways that are hard to reason about. The conservative approximate stage also leaves real duplicates in the store, which has to be accepted explicitly rather than discovered later as a quality problem.

Merge into a canonical item that keeps its sources

Problem. Picking one copy to survive and deleting the rest throws away the parts of the discarded items that the survivor lacks: an earlier first-seen date, a different provenance, a stronger confirmation, a reference some other item depends on.

Approach. Make the merge produce a canonical item that carries a list of contributing source identifiers, the earliest and latest observation times across the group, and the union of their provenance. The non-canonical copies become redirects rather than disappearing, so existing references resolve to the canonical item and nothing dangles. Repetition count is then available as a first-class signal, and the merge is reversible because the inputs are still addressable.

Trade-off. Redirects accumulate and never shrink the row count, so the storage saving that motivated deduplication is largely forfeited; only the retrieval benefit remains. Carrying the union of provenance also means the canonical item inherits the weakest source alongside the strongest, and downstream consumers must decide what to do with a mixed-origin claim.

Never compare across a scope boundary

Problem. Deduplication is a similarity operation over the whole store, and the most similar item to one person's note is frequently another person's identically-worded note. A merge across that boundary silently moves one person's memory into another's history.

Approach. Make the ownership scope part of the candidate-generation key, not a filter applied afterwards, so items from different owners are never in the same comparison set. Run the sweep per scope. Where a legitimately shared store exists, treat the shared space as its own scope rather than as a superset, so a personal item and a shared item are never candidates for merging with each other.

Trade-off. Per-scope sweeps lose the efficiency of a single global pass and scale with the number of owners rather than with the number of items, which is much worse for stores with many small owners. Genuinely shared duplicates across scopes also go undetected, so a fact known to a thousand users is stored a thousand times by design.

Quarantine before destruction

Problem. A deduplication rule is only validated by running it on real data, and by then the items it merged wrongly are gone. Tuning the threshold afterwards cannot recover them.

Approach. Make the sweep's first phase produce proposed merges rather than performing them, retained in a quarantine with their scores, their tier, and both texts. Sample from the proposals for review, measure precision at each threshold against that sample, and only then apply. After application, retain the merge records long enough that a precision problem discovered in production can be reversed in bulk rather than item by item.

Trade-off. Quarantine adds latency between duplicates arriving and duplicates being removed, so the retrieval problem persists for the length of the review cycle, and review capacity becomes the limiting factor on how much deduplication happens at all. Retaining merge records also keeps copies of content that users may reasonably believe has been consolidated away.