RAG and Knowledge Engineering
Incremental Indexing — Stack
Incremental indexing › Stack
Technology landscape as of 2026-10-10. Knowledge cutoff May 2026.
Selection criteria
The index engine's update semantics dominate this decision, and they are rarely the thing a vector or search engine is evaluated on.
- Update and delete semantics. Whether an update is a true in-place replacement or an insert plus a tombstone, and whether a delete is visible to queries immediately.
- Tombstone accumulation and compaction control. What fraction of deleted entries the engine tolerates, what triggers compaction, and whether it can be run without disrupting queries.
- Delete-by-query support. Whether all units of a document can be removed by an attribute predicate, or only by enumerating identifiers you must have stored.
- Idempotent writes. Whether writing the same unit twice is harmless, which determines how much machinery the delivery path needs.
- Alias or indirection support, so a rebuilt index can be swapped atomically and reverted.
- Partial attribute updates. Whether entitlement or freshness markers can be changed without rewriting and re-encoding the unit.
- Bulk versus streaming write throughput, which differ by an order of magnitude and determine whether a catch-up is feasible.
- Reconciliation affordances. Whether the full set of stored identifiers can be enumerated efficiently for comparison against the source of truth.
Options
| Option | Best for | Watch out for |
|---|---|---|
| Vector engines with explicit upsert and delete-by-filter (Qdrant, Weaviate, Milvus) | Document-granular updates driven by attribute predicates | Compaction behaviour under heavy churn varies sharply; measure recall between compactions |
| Search engines with segment-based indexing (Elasticsearch, OpenSearch, Apache Solr, Lucene directly) | Mature update, merge and alias mechanics plus delete-by-query | Deleted documents persist in segments until merge; merge pressure competes with query load |
| Relational stores with vector extensions (PostgreSQL with pgvector) | Transactional consistency between units, attributes and their source records | Index maintenance on large tables is the bottleneck; vacuum behaviour becomes a retrieval concern |
| Immutable index generations rebuilt into object storage | Corpora that change in large periodic batches rather than continuously | Latency between batches is the freshness floor; no path to minute-level updates |
| Log-structured change feeds driving the index (Kafka topics keyed by document identity) | Replayability, ordering per document, and rebuilding from the log | Retention on the log must exceed your worst recovery window or replay is impossible |
| Table formats with change-data capture (Apache Iceberg, Delta Lake) as the staging layer | Computing unit-level differences before touching the index | Another system to operate; small-file behaviour punishes high-frequency per-document writes |
| Managed indexers in retrieval platforms | Small corpora where update semantics can be taken on trust | Reconciliation and compaction are opaque; drift is undetectable from outside |