RAG and Knowledge Engineering
Ingestion Pipeline — Stack
Ingestion pipeline › Stack
Technology landscape as of 2026-10-10. Knowledge cutoff May 2026.
Selection criteria
The decision is about durability, replay and visibility over a long-running population of work items, not about what the individual steps do.
- Per-item durability. Whether the state of a single document survives a crash, and whether it can be queried — "where is this document" must be answerable without reading logs.
- Idempotency support. Deduplication keys, exactly-once sinks, or at least a transactional outbox, because every retry otherwise risks a duplicate indexed unit.
- Ordering guarantees per key. Whether updates and removals for one document can be forced into order without serialising the whole stream.
- Heterogeneous step cost. The ability to run a millisecond step and a ten-minute step in the same flow without the slow one monopolising the workers for the fast one.
- Backfill alongside steady state. Priority classes, separate worker pools, or rate shaping so a bulk reprocess cannot starve incremental updates.
- Artifact persistence. Whether intermediate outputs are naturally stored and addressable, or whether the framework encourages recomputing from the source.
- Operational surface. Backlog age rather than backlog depth, per-stage failure counts, and a dead-letter you can inspect and selectively replay.
- Cost of a partial redeploy. Whether a code change to one step requires draining everything in flight.
Options
| Option | Best for | Watch out for |
|---|---|---|
| Durable workflow engines (Temporal, Restate, AWS Step Functions) | Per-document state machines with long, uneven step durations and precise retry control | A workflow per document at tens of millions of documents is a scale question in its own right |
| Stream processors (Apache Flink, Kafka Streams, Apache Beam) | Continuous high-volume flows where per-key ordering and backpressure are first-class | Long external calls per record fit awkwardly; state backend sizing becomes its own operational job |
| Task queues (Celery, Sidekiq, RQ, cloud queue plus workers) | Getting a working pipeline quickly with ordinary application code | Ordering, idempotency and dead-lettering are all yours to build; most outages here start as a missing one |
| Batch orchestrators (Apache Airflow, Dagster, Prefect) | Scheduled reprocessing, backfills and jobs whose unit is a partition rather than a document | Per-document retry granularity is poor; a single bad file fails a whole task |
| Data-lake table formats as the staging layer (Apache Iceberg, Delta Lake) | Keeping extracted and segmented artifacts queryable, versioned and cheaply re-readable | Adds a second storage system to operate; small-file behaviour punishes per-document writes |
| Managed ingestion inside a retrieval platform (vendor indexer pipelines, LlamaCloud-style services) | Small corpora and teams who want no pipeline at all | The state machine is opaque; reprocessing policy and failure visibility are whatever the vendor exposes |
| Hand-rolled scripts over object storage | Prototypes and one-off corpus builds | Works until the first partial failure; there is no road from here to incremental operation |