Retrieval Evaluation

What it is

Measuring whether the retrieval stages put the right material in front of whatever consumes it. That requires a set of queries with judgements about which passages are relevant to each, metrics computed at the depths the system actually uses, and attribution of a result to the stage that produced it — first-stage recall at candidate depth, ordering quality at the cut depth, and the contribution of each arm and transformation in between.

Why it is not end-to-end answer evaluation

The question here is answerable without generating anything: was the evidence present, and was it ordered well. That independence is the point. It separates a retrieval defect from a generation defect, it is cheap enough to run on every change to a chunking rule or an index parameter, and it is stable in a way that judgements about prose are not. Assessing whether the final response was correct, useful or well written — and the judge methodology, harness design and release gating that go with it — is a different discipline applied to a different artifact, and it cannot tell you which of six retrieval stages caused a regression.

Why it matters

Every tuning decision in a retrieval pipeline — unit size, overlap, candidate depth, fusion weights, reranking depth, index parameters — is a trade with no obvious direction, and without measurement they are made by intuition and defended by anecdote. Worse, the most consequential retrieval failure is invisible by construction: when the answer was never retrieved, the system produces a fluent response from the next-best material and nothing registers an error. Recall against a judged set is the only instrument that sees that case at all.

Key decisions

• Where judgements come from. Human annotation is expensive and is the only source that can judge material the current system never retrieves; production signals are free and biased toward what was shown; model-generated judgements are cheap and inherit the biases of the model that made them.
• How the query set is sampled. A set drawn from the current system's successes measures regression and cannot discover blind spots; a set written from the domain measures capability and may not resemble traffic.
• Which depths the metrics are computed at. The numbers are only actionable if they match the real candidate depth and the real cut depth, and a metric at an arbitrary depth measures a system nobody is running.
• How unjudged passages are treated. In any realistic set most retrieved passages have no judgement, and scoring them as irrelevant systematically penalises a candidate that retrieves something new and good.
• Whether results are reported per query class — identifiers, multi-hop, conversational follow-ups, non-English — since an aggregate hides a class that fails completely.
• How the judged set survives a changing corpus, given that the document a judgement points at can be edited, re-segmented or deleted.
• Whether the evaluation runs against a frozen fixture index or the live one, which is a trade between reproducibility and relevance.

Failure modes

• The judged set was built by annotating what the current system returned, so by construction it contains nothing the system misses, and every candidate change scores as a regression.
• Metrics are reported at a depth the pipeline does not use — ten, because that is the default — while retrieval fetches a hundred and the cut passes five.
• A corpus update deletes or rewrites judged documents, the score falls, and a week is spent investigating a retrieval change that did not happen.
• The set has fifty queries, the difference between two configurations is within noise, and a decision is made on it anyway because the number moved.
• Aggregate recall is strong and queries containing part numbers fail completely, because they are four percent of the set and nobody reports by class.
• Judgements are generated by the same family of model that produces the embeddings, and the evaluation systematically prefers the configuration that model already favours.
• Evaluation runs against a fixture index that has been complete and current since the day it was built, while production has missing units and a month of lag.
• Only the final set is measured, so a regression in fusion, reranking or filtering is indistinguishable from a regression in the first-stage index.

Related capabilities

• Reranking — the depths this measurement exists to justify
• Hybrid Search — fusion rules and per-arm weights cannot be set without it
• Chunking Strategies — a segmentation change can only be compared here
• Citation and Grounding — per-answer attribution, as opposed to population measurement
• Knowledge Versioning — a comparison between runs is only valid against a known corpus state

Applies: Production readiness · Design questions