Incremental Indexing — Patterns

Difference the unit set, do not replace the document

Problem. Treating any document change as "remove everything, write everything" makes the cost of an edit proportional to the document rather than to the edit. On a corpus of long documents with frequent small revisions, that is most of the encoding bill, and it churns the index needlessly.

Approach. Compute the new unit set with content-derived identities, compare it against the identities currently stored for that document, and issue writes only for identities that are new and removals only for identities that have disappeared. Unchanged units are not touched at all, so their vectors, their positions in the index and the citations pointing at them all survive.

Trade-off. This requires the pipeline to hold the current unit identities per document, which is a second store to keep consistent with the index, and any inconsistency between them produces either orphans or missing content. It also puts a hard requirement on segmentation stability: a rule that shifts boundaries when text changes nearby defeats the whole mechanism.

Monotonic revision stamps to resolve out-of-order writes

Problem. Updates and removals for one document travel through queues that guarantee neither ordering nor exactly-once delivery. A removal applied after a subsequent update deletes live content; an older version applied after a newer one silently reverts the corpus.

Approach. Attach a monotonically increasing revision — the source's version, a sequence number from the change log, or an ingestion timestamp of sufficient resolution — to every write and removal, store it on each unit, and have the index reject any operation whose revision is not greater than the one already recorded. Late and duplicate messages become no-ops instead of corruption.

Trade-off. Not every engine can express a conditional write, and where it cannot, the check has to be done with a read first, which is neither atomic nor free. Sources that do not provide a usable version force a locally minted one, and a local clock across several workers is not a reliable order.

Rebuild into a new generation, never in place

Problem. Some changes — a new segmentation rule, a different encoder, a schema change — invalidate every unit. Applying them in place leaves the index mixed for the duration, and reverting is not possible once the old content has been overwritten.

Approach. Treat the index as an immutable generation behind an alias. Build the new generation from the stored intermediate artifacts, apply the live change stream to both while it builds, verify unit counts against the source of truth and quality against a judged query set, then move the alias. The previous generation remains until the new one has served real traffic.

Trade-off. Peak storage is the sum of the generations, dual-writing is an extra path that can itself diverge, and the verification step is the one most likely to be skipped under schedule pressure. Corpora large enough to make this expensive are also the ones for which it matters most.

Standing reconciliation between sources and index

Problem. Individual write failures are rare and permanent. Each one leaves the index slightly wrong, nothing reports it, and after a year the accumulated drift is a few percent of a corpus that everyone treats as complete.

Approach. Periodically enumerate the unit identities the pipeline believes should exist for a partition of the corpus and compare them with what the index actually holds. Missing identities are re-enqueued; unexpected ones are candidates for removal after being checked against pending deletions. Report both counts continuously, so drift is a visible number rather than the conclusion of an investigation.

Trade-off. Full enumeration is expensive enough at corpus scale that it has to be partitioned and spread over time, so systematic failures are caught within hours and isolated ones within weeks. Acting automatically on unexpected identities is also risky: an in-flight write looks exactly like an orphan.