Chunking Strategies — References

Passage segmentation as a retrieval problem, and the evidence that the retrieved unit's size and position change what a model does with it.

Segmentation as a retrieval decision

• Passage Retrieval Revisited — Marcin Kaszkiel and Justin Zobel, ACM SIGIR, 1997. Arbitrary overlapping passages compared against document and paragraph units, and the finding that the unit of retrieval is itself a tuning decision rather than a property of the corpus. Cited by name.
• TextTiling: Segmenting Text into Multi-paragraph Subtopic Passages — Marti A. Hearst, Computational Linguistics, 1997. Boundary detection from lexical cohesion, the ancestor of similarity-based splitting. Cited by name.
• Dense Passage Retrieval for Open-Domain Question Answering (Karpukhin et al., 2020), arXiv:2004.04906 — the dual-encoder formulation whose fixed-length passage construction established the hundred-word passage as a default worth questioning.

What the encoder imposes

• Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks (Reimers & Gurevych, 2019), arXiv:1908.10084 — the bi-encoder construction, and the reason a unit longer than the model's sequence limit is truncated rather than rejected.

What happens to the unit after it is retrieved

• Lost in the Middle: How Language Models Use Long Contexts (Liu et al., 2023), arXiv:2307.03172 — measured degradation in the use of material placed in the middle of a long input, which bounds how much unscored context a large returned unit can usefully carry.

No verified reference is recorded for content-addressed unit identity, for overlap ratios, or for atomic-region overflow policies. Published comparisons of segmentation settings are almost always corpus-specific, and the honest answer is that this must be measured on your own judged set rather than adopted from a paper.