Ingestion Pipeline — References

The delivery, ordering and recovery semantics that determine whether a document population ends up correctly in an index.

Delivery and processing semantics

• Kafka: a Distributed Messaging System for Log Processing — Jay Kreps, Neha Narkhede and Jun Rao, NetDB workshop, 2011. The partitioned log and the per-key ordering guarantee that most document pipelines ultimately rely on. Cited by name.
• Lightweight Asynchronous Snapshots for Distributed Dataflows — Paris Carbone, Gyula Fóra, Stephan Ewen, Seif Haridi and Kostas Tzoumas, arXiv preprint, 2015 (arXiv:1506.08603). Barrier- based snapshotting, and the basis for exactly-once effects in a streaming stage. Cited by name.
• The Dataflow Model: A Practical Approach to Balancing Correctness, Latency, and Cost in Massive-Scale, Unbounded, Out-of-Order Data Processing — Tyler Akidau, Robert Bradshaw, Craig Chambers, Slava Chernyak, Rafael J. Fernández-Moctezuma, Reuven Lax, Sam McVeety, Daniel Mills, Frances Perry, Eric Schmidt and Sam Whittle, VLDB, 2015. The separation of event time from processing time, and the vocabulary for reasoning about late and out-of-order arrivals. Cited by name.

Idempotency and transactional delivery

• Life beyond Distributed Transactions: an Apostate's Opinion — Pat Helland, CIDR, 2007. The argument for entity-scoped, idempotent operations in place of distributed transactions, which is the shape a document ingestion stage ends up with. Cited by name.
• Idempotence Is Not a Medical Condition — Pat Helland, ACM Queue, 2012. Why retry safety has to be designed into the message and the sink rather than assumed. Cited by name.

Operating the running system

• Site Reliability Engineering — the published Google SRE material, for queue-age-based alerting and the argument against alerting on symptoms that fall while the problem worsens.

No verified reference is recorded for shadow index builds with alias swaps, for content-addressed stage caching keyed on configuration identity, or for backfill-versus-incremental priority shaping. These are engineering practice without published study.