Incremental Indexing

What it is

Bringing an index to a new state by applying only the differences — the units a change actually added, modified or removed — instead of reconstructing it, together with a deliberate path for the cases where reconstruction is unavoidable. Its correctness condition is simple to state and hard to achieve: the index after applying a sequence of changes must be indistinguishable from one built from scratch against the same sources.

Why it matters

Rebuilding is the only operation whose correctness is obvious, and it is also the one that costs the entire encoding bill and takes as long as it takes. Every system of consequential size therefore applies differences, and in doing so accepts that the index now has a history. Drift between the index and its sources accumulates one unnoticed partial failure at a time, and nothing about a drifted index looks wrong: it answers, it is fast, and some fraction of its content is from a version of a document that no longer exists. The cost of the alternative is what makes this worth getting right — a corpus whose rebuild takes four weeks cannot use rebuild as its recovery strategy, so the incremental path has to be the trustworthy one.

Key decisions

• How a change is detected and what its unit is: a content hash computed after extraction, a version identifier from the source, or an explicit event. The first is robust and expensive; the last is cheap and only as reliable as its delivery.
• Whether a modified document is handled as a wholesale replacement of its units or as a difference over them. Differencing requires unit identities derived from content, and repays it by making a one-paragraph edit cost one encoding instead of four hundred.
• How ordering is enforced for a single document, since a removal that arrives after a later update resurrects or destroys content depending on which lands last.
• Which upstream changes force a full rebuild — a different segmentation rule, a different encoder, a schema change — and how that rebuild is executed without the index being unavailable or mixed while it runs.
• When compaction runs, given that most indexes mark removals rather than erasing them and both latency and result quality drift between compactions.
• How the system catches up after an outage without the catch-up itself becoming the outage.
• What reconciliation exists between the set of units the sources imply and the set the index holds, and how often it runs.

Failure modes

• A changed document's new units are written and its old ones are not removed, so the index holds two generations of the same passage and they compete in every retrieval.
• Unit identities are positional, a sentence is inserted near the top of a long document, and a trivial edit triggers a full re-encode of that document plus the invalidation of every citation into it.
• Change events are delivered at least once, the write path is not idempotent, and duplicates accumulate in proportion to how often the queue retries.
• A backfill and the live stream both write the same document, the backfill's older version lands second, and the index silently reverts to stale content.
• A document shrinks from forty units to thirty, the replacement writes thirty and the removal covers only what it wrote, and ten orphans remain retrievable indefinitely.
• The query path names a concrete index rather than an alias, so the rebuild swap is a manual reconfiguration that happens under pressure.
• Catching up after a six-hour outage saturates the encoder quota, and live updates queue behind the backlog so the outage's effect on freshness long outlasts the outage.
• Nothing ever compares the index against its sources, so a two percent drift accumulated over a year is discovered only when someone asks why a specific document is missing.

Related capabilities

• Ingestion Pipeline — the orchestration whose output this applies to the index
• Chunking Strategies — unit identity decides whether differencing is possible
• Vector Database — where tombstones accumulate and compaction has to happen
• Knowledge Freshness — the target this mechanism exists to meet
• Data Deletion Propagation — removal correctness is this mechanism's hardest half

Applies: Production readiness · Design questions