Memory and State Management
Memory Deduplication — Stack
Memory deduplication › Stack
Technology landscape as of 2026-10-10. Knowledge cutoff May 2026.
Selection criteria
You are choosing a similarity or identity mechanism whose false positives destroy information irreversibly, so the asymmetry between its two error types governs everything.
- False-positive cost. A wrongly merged pair loses a distinct fact permanently. Mechanisms must be evaluated on precision first and recall second, which is the opposite of how search is usually tuned.
- Threshold stability. Whether the score has a stable meaning across content types and over time, or whether a tuned cut-off silently drifts as the store's content changes.
- Candidate generation cost. Comparing every pair is quadratic. What blocking, bucketing, or index structure reduces the comparison set, and what it misses.
- Scope awareness. Whether the mechanism can be constrained so that items from different owners are never compared at all.
- Explainability of a match. Whether a merge decision can be shown to a reviewer as a reason rather than as a number.
- Incremental operation. Whether the mechanism supports checking one new item against the store, or only whole-store sweeps.
- Reversibility. Whether the merge record retains enough to reconstruct the inputs if the rule turns out to have been wrong.
Options
| Option | Best for | Watch out for |
|---|---|---|
| Exact hash on normalised text (SHA-256 over a canonical form) | Catching retried writes and re-imports with zero false positives | Misses every paraphrase; normalisation choices quietly become part of the identity rule |
| Structural key on typed fields (subject plus predicate plus value) | Typed belief stores where the same claim has one canonical representation | Only works where the vocabulary is controlled; unstructured items have no key to form |
| Locality-sensitive hashing over shingles (MinHash with LSH, SimHash) | Large stores of free text where near-duplicates must be found cheaply | Tuned for document-scale text; short memory items share few shingles and match poorly |
| Embedding similarity above a threshold (any vector index) | Catching paraphrase, which is the common case in conversational memory | Semantic similarity is not identity; negation and small numeric differences are nearly invisible to it |
| Entity-resolution frameworks with blocking and scoring (Dedupe, Splink, Zingg) | Record-style memories with several comparable fields and enough labelled pairs to train on | Needs labelled training pairs and a schema; overkill for free-text notes |
| Model-adjudicated pairs over a high-recall candidate set | Final arbitration where the cost of a wrong merge justifies a model call per pair | Expensive, non-deterministic, and prone to agreeing that two things are "basically the same" when they are not |
| Probabilistic set membership for ingest-time filtering (Bloom or cuckoo filters) | Cheaply rejecting items already seen during bulk import | False positives silently drop new items; no use for near-duplicates, only for exact repeats |