Ingestion Pipeline — Patterns

Per-document state machine with explicit terminal states

Problem. When progress lives only in queue depth, the question "which documents are not answerable and why" has no answer, and a corpus with holes in it looks identical to a complete one.

Approach. Give every document a row recording its current stage, the revision being processed, the timestamp of the last transition, and a terminal state that is one of indexed, deliberately skipped, or quarantined with a reason. Every stage transition is a write to that row. Dashboards count rows by state and by age rather than counting messages, and the quarantined set is a work queue a human owns.

Trade-off. The row is a write amplification on every stage for every document, and at high volume it becomes a database of its own that needs partitioning and expiry. It also introduces a second source of truth that can disagree with the index, so a periodic reconciliation between the two is not optional.

Content-addressed stage artifacts

Problem. Changing a downstream rule should not require asking the source system for the bytes again, but pipelines that stream straight through hold nothing and must re-crawl — against a quota owned by someone else, for a change that affects only the last two stages.

Approach. Write each stage's output to object storage under a key derived from the hash of its input plus the identity of the configuration that produced it. A later stage reads by key, so an unchanged input with an unchanged configuration is a cache hit. Changing one stage's configuration changes only the keys below it, and the reprocess becomes a scan of what is already stored.

Trade-off. Storage grows with the product of corpus size and configuration history, so artifacts need an expiry policy that someone owns. The hash must cover every input that changes the output, and a forgotten input — a library version, a locale — produces stale cache hits that are extremely hard to diagnose.

Shadow index build with an alias swap

Problem. A change to segmentation or encoding has to be applied to the entire corpus, but applying it in place leaves the index in a mixed state for the duration, and the duration can be days.

Approach. Build the new corpus into a separate index while the live one continues to serve, keep the incremental stream flowing into both, verify the new index against a held-out query set and a document-count reconciliation, then move a serving alias. The old index stays in place long enough to move the alias back.

Trade-off. Peak storage and encoding cost roughly double for the duration, and dual-writing the incremental stream is an extra path that can itself diverge. Verification before the swap is the whole value of the pattern and is also the part that is routinely skipped under deadline.

Priority-class separation between backfill and live updates

Problem. Bulk reprocessing and incremental updates draw on the same workers, the same encoder capacity and the same index write path, so a reprocess silently converts a minutes-old corpus into a day-old one.

Approach. Run the two as separate consumer groups with separate worker pools and separate quota allocations, and make the backfill pool yield when the incremental backlog age crosses a threshold. Report backlog age — the age of the oldest unprocessed change — rather than depth, because depth falls while age rises during exactly this failure.

Trade-off. Reserved capacity for the incremental path is idle most of the time, and backfills take materially longer than the hardware would allow. The yielding threshold is a tuning parameter that needs revisiting whenever either workload's shape changes.