Query Rewriting — Patterns

Retrieve with the original and the rewrite, then fuse

Problem. A rewrite is a bet that replaces the only thing the user actually said. When the bet loses, the correct passages are not merely ranked lower — they are never retrieved, and no later stage can recover them.

Approach. Issue both queries, the original and the transformed one, and merge the two result sets by rank before anything downstream reads them. A good rewrite contributes passages the original missed; a bad one contributes passages that the original's results outrank. The failure mode degrades from catastrophic to merely wasteful.

Trade-off. Two searches per query, which doubles the retrieval cost and widens the latency tail, and a merged set that is more diverse and therefore less concentrated on the best material. When the rewrite is reliably better, the original's contribution is pure overhead.

Pin the user's literal constraints

Problem. Rewrites lose specifics. A model asked to clarify "what changed in the pricing policy for enterprise customers in March" will cheerfully return a cleaner query about the pricing policy, and the retrieval that follows is about the wrong three things.

Approach. Extract the literal constraints from the original query first — dates, numbers, quoted phrases, identifier-shaped tokens, named entities — and carry them through as structured attributes and as required terms rather than leaving them to survive the rewrite. The rewrite operates on the remaining prose, and the constraints are reattached to the search as predicates and as lexical requirements.

Trade-off. Extraction has its own error rate, and a constraint pinned in error becomes a hard filter that returns nothing. Over-pinning also defeats the purpose: a date the user mentioned in passing becomes a requirement the corpus cannot satisfy.

Resolve against a bounded, recent window

Problem. Giving a rewriter the whole conversation lets it resolve references correctly and also lets it drag an abandoned topic into a new question. Longer history makes resolution better and contamination worse at the same time.

Approach. Pass a small, recent window of turns, and give the rewriter an explicit option to decide that the current query is self-contained and leave it untouched. Treat a rewrite that introduces entities absent from both the current query and the window as a signal to fall back to the original.

Trade-off. A genuinely long-running thread where the subject was established twenty turns ago will resolve incorrectly or not at all, and the window size that fixes it reintroduces the contamination. The self-contained option is also a judgement the model makes, with its own systematic biases.

Rewrite cache keyed on query and resolved context

Problem. Query traffic repeats heavily, and paying for a model call to produce an identical rewrite is spend on the critical path that buys nothing. Caching on the raw query alone is wrong, because the same words in two conversations resolve differently.

Approach. Key the cache on the normalised original query together with a digest of the context actually used to resolve it, so an identical standalone query is a hit and an identical follow-up in a different conversation is a miss. Invalidate on changes to the rewriting prompt or model, which means the prompt's identity has to be part of the key.

Trade-off. Hit rates are good for standalone queries and poor for conversational ones, which are precisely the queries where rewriting is most expensive and most valuable. Including the context digest also makes the cache nearly useless if the digest is computed over too much history.