Query Decomposition

What it is

Answering one question with several retrievals because no single retrieval can satisfy it: comparisons that need evidence about two entities, multi-hop questions where the second search cannot be formulated until the first has returned, aggregations over a set whose members must be enumerated first, and compound requests that bundle three unrelated asks into one sentence. The substance is a small retrieval plan — what to ask, in what order, within what budget — and a rule for recombining what comes back.

Why it is not query rewriting

Reformulating produces one better query and one result set, and if it goes wrong the result set is simply about something else. Splitting changes the arity of the operation: there is now a plan, a fan-out, a budget that multiplies, and a join. The join is where this capability's distinctive errors live — a missing sub-result silently weakening a comparison, duplicate passages across branches consuming the budget, sub-answers about different time periods being set against each other as though they were comparable. None of those failures exist when there is only one retrieval.

Why it matters

A class of perfectly ordinary questions cannot be answered by any single search, however good. "Which of our three regional policies has the shortest notice period" has no passage that contains the answer, and a top-k retrieval over the obvious terms returns the three policies' introductions. Without decomposition the system answers these with whichever document matched best, which looks like a reasonable answer and is an arbitrary one. With it, the system can be wrong in more interesting ways — and the cost, latency and context consumption grow with the fan-out, so the budget is as much a part of the design as the plan.

Key decisions

• Whether sub-queries are produced all at once or discovered as the chain proceeds. A fixed plan runs in parallel and cannot follow a dependency; an iterative plan follows the chain and pays for each hop serially.
• Who decides that a question needs splitting, and what the cost of a false positive is — a simple lookup decomposed into four retrievals is slower, more expensive and usually worse.
• Hard budgets on breadth and depth: maximum branches, maximum hops, maximum total passages admitted. Without them the failure is unbounded rather than degraded.
• How sub-results are recombined: concatenated into one context, summarised per branch and then compared, or reduced to structured values before any comparison is attempted.
• Whether sub-queries inherit the parent's constraints — entitlement predicate, date range, tenant, language — given that losing a constraint on a branch is both a quality and a security defect.
• What an empty branch means. Treating "no evidence found for this entity" as equivalent to "nothing to report" turns a gap into a false assertion of completeness.
• Whether the plan is recorded, since a wrong answer here is usually a wrong plan rather than a wrong passage.

Failure modes

• A comparison splits into per-entity retrievals that return passages from different years, and the answer presents a difference that is entirely an artifact of the dates.
• One branch returns nothing, the gap is dropped silently during recombination, and the answer asserts a complete comparison across items it had evidence for only some of.
• Fan-out multiplies the retrieved material until the combined context is larger and noisier than a single well-targeted retrieval would have been, and the answer is worse than the undecomposed baseline.
• An iterative chain never terminates on its own because each hop produces a plausible next question, and only a hop limit stops it.
• The parent's date or tenant constraint is expressed in natural language, so the sub-queries generated from it drop the constraint and retrieve across the whole corpus.
• The same passage is retrieved by three branches and occupies three slots in the final context, reading to the model as independent corroboration.
• Decomposition is applied to a question that one passage answered exactly, and the extra retrievals introduce adjacent material that pulls the answer off target.
• Each branch is individually well evidenced and the final synthesis asserts a relationship between them that no retrieved passage supports.

Related capabilities

• Query Rewriting — the single-retrieval alternative, often the better one
• Citation and Grounding — a synthesised comparison must still attribute each half
• Reranking — applied per branch or after the join, with different consequences
• Retrieval Evaluation — per-branch recall is measurable; the join is what needs end-to-end judgement
• Task Decomposition — the same idea applied to an agent's work rather than to evidence gathering

Applies: Production readiness · Design questions