Retrieval Evaluation — Patterns

Pool candidates from several configurations before judging

Problem. Judging only what the current system returns builds a set that cannot contain anything the current system misses. Every alternative configuration then appears worse, because the relevant passages it found are unjudged and scored as irrelevant.

Approach. For each query, gather the top results from several meaningfully different configurations — lexical only, dense only, with and without reranking, different segmentations — take the union, and judge that pool. The resulting judgements cover material no single configuration would have surfaced, and a new configuration is scored against a set that was not built around its competitor.

Trade-off. The pool is much larger than any single result list, so annotation cost rises roughly with the number of configurations pooled, and the set still only covers what some configuration in the pool found. Pools also go stale: a genuinely new retrieval method reintroduces the original bias until the pool is extended.

Metrics computed at the depths the system runs at

Problem. Default metric depths are conventions, not descriptions of any particular pipeline. A number reported at depth ten says nothing useful about a first stage that fetches two hundred candidates for a reranker, and a recall figure at the candidate depth says nothing about what the user sees.

Approach. Derive the measurement plan from the pipeline: recall at the first-stage candidate depth, because its only job is to not lose the answer; ordering quality at the reranker's cut depth, because that is what reaches the consumer; and both recomputed whenever either depth changes. Report the depth in the name of every number so a comparison between two runs cannot silently compare different things.

Trade-off. Metrics tied to your configuration cannot be compared with published results or with another team's numbers, which removes a useful sanity check. Changing a depth also breaks continuity with historical runs, so each change needs a period where both are reported.

Query classes as first-class slices

Problem. A retrieval system's failures are concentrated in populations that are small in the evaluation set and large in the consequences — exact identifiers, acronyms, conversational follow-ups, negations, languages other than the dominant one. An average over all queries is dominated by the easy majority and moves hardly at all when one of these classes breaks completely.

Approach. Label every query in the set with its class when the set is built, require each class to have enough queries to be measurable rather than merely represented, and report per class with the aggregate presented as a summary of the slices rather than as the headline. Treat a class that regresses as a blocking result even when the aggregate improved.

Trade-off. Maintaining balanced classes makes the set unrepresentative of traffic, so the aggregate ceases to estimate production quality and becomes a weighted artifact. Class definitions also drift as the product changes, and re-labelling an existing set is tedious enough that it usually does not happen.

Bind every run to a corpus revision

Problem. Scores move when the corpus changes, and corpus changes are continuous. A comparison between last month's run and today's attributes to the retrieval configuration a difference that was caused by documents being added, edited or removed.

Approach. Record the corpus state identifier with every evaluation run, and for comparisons that must isolate the configuration, evaluate both against the same frozen state. Keep a separate, scheduled run against the live corpus to detect drift that is caused by content rather than by code, and report the two separately rather than averaging them into one trend line.

Trade-off. Frozen states must be retained, which costs storage and goes stale, and a configuration tuned on a frozen corpus can be tuned to its peculiarities. The two-track reporting also doubles the number of charts, and teams tend to watch only one of them.