Reranking — Patterns

Set both depths from the measured recall and precision curves

Problem. Input depth and output depth are usually inherited from an example, and they are the two parameters that determine the stage's entire cost and most of its benefit. Tuning them by intuition produces either a stage that cannot help, because the answer is not in the candidate set, or one that is paying to score passages that were never going to be used.

Approach. Measure first-stage recall as a function of depth on a judged set and take the input depth from where the curve flattens — beyond that point the stage is scoring passages that contain nothing. Measure end-to-end answer quality as a function of output depth and take the output from where that curve flattens. Record both numbers with the measurement that justified them, and re-measure when either model changes.

Trade-off. The curves are averages over a query population whose classes behave differently, so the single pair of numbers is already a compromise, and queries with diffuse evidence are systematically underserved by a depth chosen for the median. Re-measuring also requires a judged set that is maintained, which is a standing cost.

Diversity constraint over the reordered set

Problem. Optimising precision at the top produces a set of passages that are all correct and all the same: five windows over one section of one document, with the second-best source absent entirely. The consumer sees apparent corroboration and no coverage.

Approach. Apply a constraint while selecting the output set rather than taking the top scores directly — a cap on passages per source document, or a selection rule that penalises similarity to what has already been chosen. Score first, then select under the constraint, so that relevance still drives the ordering and diversity only breaks ties against redundancy.

Trade-off. When a single document genuinely is the answer, capping its contribution replaces relevant material with less relevant material from elsewhere, and the cap is a blunt instrument for distinguishing the two cases. Similarity-penalised selection also needs a similarity measure, which is another model or another threshold.

A calibrated floor that lets the system say nothing was found

Problem. A ranking stage always returns a best candidate, including for questions the corpus cannot answer. Downstream, the best of a bad set is indistinguishable from a good match, and the product answers confidently from irrelevant material.

Approach. Calibrate the score distribution on a judged set that deliberately includes unanswerable queries, pick a floor from the resulting separation, and have the stage return an empty set rather than a weak one when nothing clears it. Emit the decision as an explicit signal so the consumer can respond differently rather than inferring from an empty list.

Trade-off. The floor is a property of the model and the first stage together, so it silently expires whenever either changes, and an uncalibrated floor is worse than none because it creates false confidence in both directions. Products also resist returning nothing, and the threshold tends to drift downward under that pressure until it stops doing anything.

Bypass path exercised as a normal mode

Problem. A stage on the critical path with an external dependency will be unavailable at some point. If the only tested behaviour is the healthy one, the outage becomes a retrieval outage, and if the fallback is untested it fails on first use.

Approach. Treat passing the first-stage ordering through as a supported operating mode, not an error path: give it a timeout that fires well inside the request budget, route a small share of traffic through it continuously so its quality is measured rather than assumed, and surface which mode served each request as a request attribute.

Trade-off. Running a share of traffic on the degraded path means deliberately serving worse results to some users, and the share that makes the measurement meaningful is larger than the share that feels acceptable. A timeout tight enough to protect the budget will also occasionally discard work that was about to complete.