Chunking Strategies — Patterns

Index the span, return the section

Problem. The span that matches a query most precisely is usually smaller than the span a model needs in order to use it. Optimising the unit for one sacrifices the other.

Approach. Encode and index small spans — a few sentences, a single clause — but store for each one a reference to the larger region it came from. At retrieval the small span is what is scored and deduplicated; the parent section is what is fetched and handed to the answer layer, with adjacent siblings merged when several children of the same parent are retrieved.

Trade-off. The returned text is larger than the matched text, so budget is consumed by material that was never scored for relevance, and a long parent can crowd out another document entirely. It also requires a second store and a merge rule, and the merge rule is where duplicates reappear.

Boundary rules derived from the document's own structure

Problem. A fixed length is an arbitrary cut that is wrong wherever the document had a real boundary, and a document that was written with headings has already told you where its units are.

Approach. Walk the parsed element tree and emit a unit per leaf section, splitting further only when a section exceeds the encoder's limit and merging adjacent sections only when they fall below a floor. Carry the full heading path onto each unit as a prefix so that a unit titled "Limitations" is distinguishable from the ninety others with that heading.

Trade-off. Section sizes in real corpora vary by two orders of magnitude, so the resulting units are heterogeneous and the retrieval scores across them are not comparable in the way equal-length units are. Documents with no heading structure fall back to the arbitrary cut anyway, so both paths must be maintained.

Content-addressed unit identity

Problem. Identifying units by their ordinal position means a one-sentence insertion near the top of a document renames every unit below it. The index sees a mass deletion and a mass insertion, the re-encode cost is proportional to the document rather than to the edit, and every existing citation into that document breaks.

Approach. Derive each unit's identifier from a hash of its own normalised text together with the document's stable identity, not from its position. Updating a document becomes a set difference: units whose text is unchanged keep their identifiers and need no work, and only the genuinely new units are encoded and written.

Trade-off. Two identical units in the same document — a repeated boilerplate clause, a duplicated row — collide unless the hash includes a disambiguator, and a disambiguator derived from position reintroduces the original problem for those units. Normalisation choices also become load-bearing: a whitespace change that the normaliser does not absorb produces a full invalidation.

Atomic regions with an explicit overflow policy

Problem. Some regions are meaningless when split — a table and its header, a code block, a clause and its sub-clauses — and some of them are larger than the encoder's limit, so "never split" is not implementable as stated.

Approach. Mark such regions atomic during parsing and have the splitter treat them as single units regardless of length. When an atomic region exceeds the limit, apply a declared overflow policy rather than the default cut: repeat the table header on each part, repeat the clause number, or store the region whole in the document store and index a generated summary of it that points back to the whole.

Trade-off. Repeated headers inflate the corpus with near-duplicate text, and indexing a summary in place of the content means the index no longer contains the words a user might search for. Both options trade a known, bounded loss for an unbounded one, which is the point, but neither is free.