RAG and Knowledge Engineering

RAG and Knowledge Engineering

Retrieval Evaluation — References

Retrieval evaluation › References

72 of 81 in this book · 2 min read

Metrics, judged-set construction, and the known hazards of incomplete judgements and model-generated labels.

Metrics

  • Cumulated Gain-based Evaluation of IR Techniques — Kalervo Järvelin and Jaana Kekäläinen, ACM Transactions on Information Systems, 2002. Discounted cumulative gain and its normalisation, including graded relevance and why the cut depth must be stated with the number. Cited by name.
  • Introduction to Information Retrieval — Christopher D. Manning, Prabhakar Raghavan and Hinrich Schütze, Cambridge University Press, 2008. Chapter 8 covers precision, recall, mean average precision and the statistical caveats on comparing systems on small query sets. Cited by name.
  • trec_eval — the reference implementation of the standard metrics, and the definition that any other implementation should be checked against.

Building the judged set

  • Overview of the TREC 2019 Deep Learning Track — Nick Craswell, Bhaskar Mitra, Emine Yilmaz, Daniel Campos and Ellen M. Voorhees, TREC, 2020. Pooling across participating systems, the judgement process, and the separation of full-ranking from reranking subtasks. Cited by name.
  • Bias and the Limits of Pooling for Large Collections — Chris Buckley, Darrin Dimmick, Ian Soboroff and Ellen Voorhees, Information Retrieval, 2007. Evidence that pooled judgements systematically favour the systems that contributed to the pool. Cited by name.
  • Retrieval Evaluation with Incomplete Information — Chris Buckley and Ellen M. Voorhees, ACM SIGIR, 2004. The paper that introduced bpref, a measure designed for the realistic case where most retrieved passages are unjudged, rather than treating unjudged as non-relevant. Cited by name.
  • MS MARCO: A Human Generated MAchine Reading COmprehension Dataset — Payal Bajaj, Daniel Campos, Nick Craswell, Li Deng, Jianfeng Gao, Xiaodong Liu, Rangan Majumder, Andrew McNamara, Bhaskar Mitra, Tri Nguyen, Mir Rosenberg, Xia Song, Alina Stoica, Saurabh Tiwary and Tong Wang, NIPS workshop, 2016. The large sparse-judgement collection behind most retrieval training, and a cautionary example of single-judgement-per-query evaluation. Cited by name.

Benchmarks and transfer

  • BEIR: A Heterogeneous Benchmark for Zero-shot Evaluation of Information Retrieval Models (Thakur et al., 2021), arXiv:2104.08663 — zero-shot retrieval across domains, and the measured unevenness of transfer that makes an in-domain judged set necessary.
  • MTEB: Massive Text Embedding Benchmark (Muennighoff et al., 2022), arXiv:2210.07316 — the task taxonomy and the per-task reporting that the aggregate average obscures.

Model-generated judgements

  • Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena (Zheng et al., 2023), arXiv:2306.05685 — the measured biases of model judges, including position and verbosity effects, and the case for validating a judge against human labels before relying on it.
  • RAGAS: Automated Evaluation of Retrieval Augmented Generation — Shahul Es, Jithin James, Luis Espinosa-Anke and Steven Schockaert, 2023. Reference-free metrics for retrieval-augmented systems, including context precision and recall computed with a model judge. Cited by name.

No verified reference is recorded for binding evaluation runs to a corpus revision, or for query-class slice design in production retrieval evaluation. These are engineering practice rather than published results.