RAG and Knowledge Engineering

RAG and Knowledge Engineering

Incremental Indexing — References

Incremental indexing › References

64 of 81 in this book · 2 min read

Index update mechanics, the delivery guarantees an update path depends on, and the specific difficulty of updating a graph-based vector index.

Index maintenance

  • Introduction to Information Retrieval — Christopher D. Manning, Prabhakar Raghavan and Hinrich Schütze, Cambridge University Press, 2008. Chapter 4 covers dynamic indexing, logarithmic merging and the treatment of deletions as an invalidation list, which is still how segment-based engines behave. Cited by name.
  • The Log-Structured Merge-Tree (LSM-Tree) — Patrick O'Neil, Edward Cheng, Dieter Gawlick and Elizabeth O'Neil, Acta Informatica, 1996. The write-optimised structure and its compaction behaviour, which is the shape most index and vector stores adopt. Cited by name.
  • Apache Lucene — the segment and merge model behind most search engines, and the authority on how deletions persist until a merge reclaims them.

Updating an approximate vector index

  • Efficient and Robust Approximate Nearest Neighbor Search Using Hierarchical Navigable Small World Graphs (Malkov & Yashunin, 2016), arXiv:1603.09320 — the graph construction, and therefore the reason removing a node is not a local operation and why deletions are usually marked rather than applied.
  • FreshDiskANN: A Fast and Accurate Graph-Based ANN Index for Streaming Similarity Search — Aditi Singh, Suhas Jayaram Subramanya, Ravishankar Krishnaswamy and Harsha Vardhan Simhadri, arXiv preprint, 2021. An index designed for continuous insertion and deletion, and an explicit account of the recall degradation that untreated updates cause. Cited by name.

Delivery guarantees the update path rests on

  • Life beyond Distributed Transactions: an Apostate's Opinion — Pat Helland, CIDR, 2007. The case for entity-scoped idempotent operations, which is what makes a replayed index write safe. Cited by name.
  • Kafka: a Distributed Messaging System for Log Processing — Jay Kreps, Neha Narkhede and Jun Rao, NetDB workshop, 2011. Per-key ordering in a partitioned log, and log retention as the bound on how far back an index can be rebuilt by replay. Cited by name.

No verified reference is recorded for unit-level differencing against stored identities, for revision-stamped conditional index writes, or for source-to-index reconciliation cadence. The index literature covers the data structures and stops short of the bookkeeping around them.