Ingestion Pipeline

What it is

The control plane that carries a document from the moment a source emits it to the moment it is answerable, and that knows at any instant where every document in flight has got to. It sequences the transformations, decides what is retried and what is quarantined, keeps the whole thing idempotent under replay, and makes the population of documents — not any individual one — an observable object with counts, lags and backlogs.

Why it is not any one of its stages

Extraction, segmentation and encoding each have their own quality question, and each can be judged by looking at one document. The pipeline's questions are only visible across a population: are updates and deletes for the same document applied in order, does replaying yesterday's batch change the index, what happens to the eleven documents that failed, and can a configuration change be rolled out to forty million documents without taking retrieval down. A pipeline can be excellent while its extraction is poor, and a superb extractor inside a pipeline with no dead-letter path produces a corpus with invisible holes in it.

Why it matters

Retrieval quality incidents are usually ingestion incidents that nobody attributed. A corpus does not announce that six percent of it failed to process; it answers slightly worse, for slightly more questions, forever. The pipeline is the only place where that becomes a number. It is also where reprocessing economics are decided: whether changing a segmentation rule is an afternoon or a three-week re-crawl depends entirely on whether intermediate artifacts were kept and whether work is addressed by content rather than by position.

Key decisions

• Whether each document carries an explicit processing state that survives restarts, or whether progress is implicit in queue depth. Only the first can answer "which documents are not in the index and why".
• The idempotency key, and what it is computed over. Keying on the source identifier alone makes a replay overwrite; keying on content alone makes a permission change invisible; most systems need a composite and most discover that late.
• Which intermediate artifacts are persisted. Keeping extracted text and the segmentation output turns a re-encode into a local job; keeping nothing turns it into a re-crawl governed by someone else's quota.
• How ordering is guaranteed for a single document when its update and its removal are in flight at once, given that most queueing systems guarantee neither order nor exactly-once.
• Where the quarantine is, what goes into it, and who looks at it. A dead-letter nobody reads is a slower form of silent loss.
• How a configuration change is rolled out: in place over the live index, or into a parallel index that is swapped once it is complete and verified.
• Whether throughput is shaped so that a bulk reprocess cannot starve the incremental path, since the two compete for the same encoders, the same index and the same quotas.

Failure modes

• A removal and a subsequent re-publication of the same document are processed out of order, the removal lands last, and a live document disappears from the corpus with every stage reporting success.
• A worker crashes after writing to the index and before acknowledging the message; the retry writes a second copy under a freshly generated identifier, and both copies compete in retrieval forever.
• One pathological file — a thousand-page scan, a deeply nested archive — occupies a worker for hours, and because the pool is shared the entire backlog stalls behind it.
• Documents that produced no extractable content are counted as processed, so the dashboard shows a complete corpus and the index is missing every file of one type.
• Throughput is reported in documents per second while the thing that matters is indexed units per document, so an extraction regression that halves the recovered text passes unnoticed.
• A full reprocess is launched during business hours, saturates the encoder quota, and the incremental path falls hours behind while the dashboards show both jobs succeeding.
• Retries are unbounded and a permanently malformed document is reprocessed every few minutes for months, consuming a measurable fraction of the entire budget.
• The pipeline is restarted with a changed segmentation configuration and only new documents use it, leaving a corpus whose units were produced by two different rules with no record of which.

Related capabilities

• Data Connectors — the upstream that feeds this orchestration
• Incremental Indexing — how the index absorbs what this produces
• Document Parsing — a stage whose output this keeps, retries and quarantines
• Chunking Strategies — a stage whose configuration change forces a reprocess
• Knowledge Freshness — the end-to-end lag this orchestration is measured by

Applies: Production readiness · Design questions