RAG and Knowledge Engineering
Retrieval Evaluation — Stack
Retrieval evaluation › Stack
Technology landscape as of 2026-10-10. Knowledge cutoff May 2026.
Selection criteria
What is being assembled is a judged set, a metric implementation and a way of running both repeatedly. The judged set is the expensive part and the one that decides whether any of it is worth anything.
- Judgement provenance and revisability. Whether a judgement records who or what made it, at what time, against which document revision, and whether it can be corrected.
- Pooling support. Whether the harness can combine results from several configurations before judging, which is the only way to escape measuring your current system's blind spots.
- Metric correctness under incomplete judgements, including metrics that distinguish unjudged from non-relevant rather than conflating them.
- Configurable depth. Whether every metric can be computed at the depths your pipeline uses, not at the library's defaults.
- Per-stage instrumentation. Whether the harness can capture and score intermediate result sets, not only the final one.
- Slice reporting by query class, language, source and document type, with significance information rather than bare means.
- Corpus-state binding. Whether a run records the corpus revision it evaluated against, so two runs can be compared honestly.
- Cost and cadence. Whether a full run is cheap enough to execute on every pipeline change, which is what makes it a tool rather than a quarterly report.
Options
| Option | Best for | Watch out for |
|---|---|---|
| Classical IR evaluation tools (trec_eval, pytrec_eval, ir_measures) | Correct, well-defined metric implementations with a long track record | Expect TREC-format inputs; they evaluate runs and have nothing to say about building the judged set |
| IR experiment frameworks (PyTerrier, ir_datasets, Anserini and Pyserini) | Reproducible pipelines, standard collections and strong lexical baselines | Oriented toward research collections; wiring them to your production retrieval takes work |
| Zero-shot retrieval benchmarks (BEIR, MTEB retrieval tasks) | Shortlisting models and sanity-checking a pipeline against known-good numbers | Public-domain corpora; good performance there does not transfer to your documents |
| RAG evaluation frameworks (RAGAS, TruLens, DeepEval, Phoenix) | Retrieval-specific metrics alongside answer metrics, with model-generated judgements | Model-judged relevance inherits model bias; validate the judge against human labels before trusting it |
| Annotation platforms (Label Studio, Argilla, Prodigy) | Producing and maintaining human judgements with an auditable trail | Annotation guidelines, not tooling, determine quality; budget for adjudication of disagreements |
| Production signal collection (click and thumbs logging, abandoned-session tracking) | Large volumes of cheap, traffic-representative preference data | Heavily biased by position and by what was shown; useless for discovering what was never retrieved |
| Experiment tracking (MLflow, Weights & Biases, or a plain results table in version control) | Keeping runs comparable over time with their configuration and corpus state | A run without its configuration and corpus revision recorded is not comparable to anything |
Models
Model-generated relevance judgements are what make a judged set affordable at the size where its numbers mean something, and they are also the fastest way to produce a confident, wrong measurement. Treat the judge as a component to be evaluated before it is used: label a few hundred query-passage pairs by hand, measure the judge's agreement with those labels, and report that agreement alongside every result it later produces. Prefer a judge from a different model family than the one behind your encoder and reranker, because a judge that shares their representations will systematically agree with what they retrieved. Be explicit about which task is being delegated — judging whether a passage is relevant to a query is a far more reliable use than inventing the queries, since a model asked to write questions from a passage produces questions that passage answers unusually well, which inflates recall for everyone. Keep the prompt, the model identifier and the revision as part of the judged set's identity, and re-check agreement when any of them changes.