Chunking Strategies

What it is

Deciding the unit of retrieval: the span of a document that is encoded, indexed, scored and returned as an indivisible whole. The choice fixes what a single relevance score is a score of, what the smallest citable region can be, and how much irrelevant text rides along with every relevant hit. It also fixes the identity of that unit, which determines how much work a small edit to a large document costs.

Why it matters

A retriever can only return units that exist. If a question's answer is spread across a boundary, no ranking function recovers it; if a unit is a whole manual, its encoding is an average of everything in it and resembles nothing in particular. The effect is asymmetric and invisible: oversized units reduce precision and inflate the budget spent on each result, while undersized units lose the context that made the text interpretable, and neither produces an error. This is also the setting most often tuned by intuition and least often measured, because changing it requires re-encoding the corpus.

Key decisions

• The unit itself: a fixed token span, a sentence or paragraph group, a structural section taken from the document's heading tree, or the whole document with splitting deferred until after encoding. Structural units match how documents are written; fixed spans match how encoders are sized.
• Overlap, and what it buys against what it costs. Overlap reduces the chance of splitting an answer and guarantees that a corpus contains multiple near-copies of its most important sentences, which then compete for the same result slots.
• Whether the indexed unit and the returned unit are the same object. Encoding a small span for precision and returning its parent section for context is a different design from either extreme and needs a parent-child relationship stored explicitly.
• Whether each unit is made self-contained — prefixed with its document title and heading path, or rewritten to resolve pronouns — so that it means something when read without its neighbours.
• Which structures must never be split: a table and its header, a code block, a numbered clause and its sub-clauses, a figure and its caption.
• How a unit's identity is derived. Positional identifiers invalidate everything after an edit; content-derived identifiers survive insertion and make incremental updates cheap.
• The ceiling imposed by the encoder's maximum sequence length, which silently truncates anything longer rather than refusing it.

Failure modes

• A fixed-length split lands in the middle of a table, so one unit holds the header row and another holds the numbers, and neither can answer a question about the table.
• Units are paragraphs, and a paragraph that begins "It must be renewed annually" is retrieved for a query about renewal with no indication of what "it" was.
• Generous overlap means the top five results are five windows over the same two hundred words, and the answer is produced from one passage presented five times.
• A policy document is indexed as a single unit because it is "one topic", and its encoding sits equidistant from every question anyone asks about it.
• Identifiers are the ordinal position of the unit in the document, so adding a sentence to the introduction changes every identifier in a five-hundred-page manual and forces a full re-encode and a full citation invalidation.
• Units exceed the encoder's sequence limit, the tail of every long section is dropped at encode time, and nothing in the pipeline reports a truncation.
• Heading context is stripped, so a hundred units across a hundred product manuals all read "Installation" and are mutually indistinguishable.
• A boundary rule tuned on prose is applied to source code and splits a function from its signature, which is the only line containing the name anyone searches for.

Related capabilities

• Document Parsing — supplies the structure that makes non-arbitrary boundaries possible
• Embedding Generation — the stage whose sequence limit caps unit size
• Incremental Indexing — unit identity decides how much a small edit costs
• Citation and Grounding — the unit is the finest region a citation can resolve to
• Retrieval Evaluation — the only way to compare two boundary policies honestly

Applies: Production readiness · Design questions