Memory and State Management

Memory and State Management

Memory Deduplication — References

Memory deduplication › References

44 of 69 in this book · 2 min read

The established techniques for deciding that two records are the same thing, and the probabilistic structures used to do it at scale.

Near-duplicate detection

  • On the Resemblance and Containment of Documents — Andrei Z. Broder, Compression and Complexity of Sequences (SEQUENCES), 1997. Shingling and the resemblance measure that near-duplicate detection is built on, including why short texts resemble each other badly. Cited by name.
  • Similarity Estimation Techniques from Rounding Algorithms — Moses S. Charikar, ACM STOC,
    1. The hashing scheme behind SimHash, which turns a similarity comparison into a cheap distance between fingerprints. Cited by name.
  • Detecting Near-Duplicates for Web Crawling — Gurmeet Singh Manku, Arvind Jain and Anish Das Sarma, WWW, 2007. Applying fingerprint-based near-duplicate detection at scale, with the blocking strategy that makes the comparison set tractable. Cited by name.

Deciding that two records are one entity

  • A Theory for Record Linkage — Ivan P. Fellegi and Alan B. Sunter, Journal of the American Statistical Association, 1969. The probabilistic framework for match decisions, including the explicit treatment of the two error types and of a review region between them. Cited by name.
  • Duplicate Record Detection: A Survey — Ahmed K. Elmagarmid, Panagiotis G. Ipeirotis and Vassilios S. Verykios, IEEE Transactions on Knowledge and Data Engineering, 2007. A survey of field-level comparison functions, blocking methods, and evaluation. Cited by name.

Cheap membership tests

  • Space/Time Trade-offs in Hash Coding with Allowable Errors — Burton H. Bloom, Communications of the ACM, 1970. The filter used to reject already-seen items during bulk ingestion, and the one-sided error that makes it safe in one direction only. Cited by name.

No verified reference is recorded for deduplication of short conversational memory items, for threshold selection over embedding similarity, or for preserving repetition counts through a merge. These are engineering practice without published study.