Query Rewriting

What it is

Transforming the text a user supplied into text that will retrieve well, before any index is consulted. The transformations are of a few recognisable kinds: resolving what a follow-up turn left implicit, normalising spelling and expanding abbreviations, widening a query with related terms, and generating a passage that looks like the answer so that the index is searched with something shaped like its own contents rather than like a question.

Why it matters

The gap between how people ask and how documents are written is where most retrieval failures live, and it is cheaper to close on the query side than by re-encoding a corpus. A three-word follow-up carries almost no retrievable signal on its own; an internal abbreviation appears in no document that spells it out; a question phrased as a question is lexically and geometrically unlike the declarative paragraph that answers it. Each of those is fixable in a few hundred milliseconds before the search runs. The same leverage is the risk: this stage changes what the system looks for, and a confident wrong rewrite produces a confident, fluent answer to a question nobody asked.

Key decisions

• Which transformations are applied, and whether they compose. Context resolution and term expansion are usually safe together; expansion stacked on a generated hypothetical passage drifts a long way from what the user said.
• Whether the rewrite replaces the original query or joins it. Retrieving with both and fusing the results costs a second search and bounds the damage a bad rewrite can do.
• What the rewriter is allowed to see. Conversation history is what makes resolution possible and is also how a stale topic from six turns ago contaminates a new question.
• Whether a model is on the critical path for every query, given that most queries do not need rewriting and all of them will pay for it.
• Whether the rewritten query is recorded and surfaced — in traces for diagnosis, and to the user where the product can afford to show what it searched for.
• What happens when the rewrite retrieves worse than the original. Without a comparison there is no detection, and without a fallback there is no recovery.
• How short, exact or already-precise queries are excluded, since rewriting an identifier is a pure loss.

Failure modes

• A pronoun is bound to the wrong antecedent in a multi-topic conversation, and the system retrieves accurately for a question the user did not ask.
• Expansion adds near-synonyms that pull the query's representation away from a specific term, and a query that would have matched exactly now matches a general discussion instead.
• A generated hypothetical passage invents a specific — a version number, a date, a product name — and retrieval anchors on the invention rather than on the user's actual constraint.
• Conversation history is passed in full, so a resolved query inherits a topic the user abandoned several turns earlier and the results are about that instead.
• The rewriter was prompted and evaluated in one language and silently degrades every other language in the product, which nobody notices because the eval set is monolingual.
• The rewrite is not recorded anywhere, so an investigation into a bad answer can see the retrieved passages and the response but not the query that actually ran.
• Every query pays an extra model call and its latency, including the majority that were already specific, and the added latency is attributed to the generation model.
• A rewrite shortens a long, detailed question into a tidy one, discarding the constraints that made the user's request specific.

Related capabilities

• Query Decomposition — the adjacent choice of issuing several retrievals instead of one better one
• Hybrid Search — each arm responds differently to the same rewrite
• Embedding Generation — the query encoding this stage feeds, with its own prefix conventions
• Retrieval Evaluation — where a rewrite's effect is measured rather than assumed
• Conversation Memory — the source of the context that makes resolution possible

Applies: Production readiness · Design questions