RAG Evaluation — References

Material on the architecture these measurements are taken over, on judging support against supplied evidence, and on what the position of a passage in the context does to whether it gets used.

The architecture

• Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks (Lewis et al., 2020), arXiv:2005.11401 — the original argument that a retrievable external store can carry knowledge the parameters do not, which is what makes the separation of support from correctness necessary in the first place.

Judging support

• Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena (Zheng et al., 2023), arXiv:2306.05685 — the biases that apply to any judge asked whether a passage supports a claim, and the case for validating such a judge against human verdicts before trusting its aggregates.
• The attributed-question-answering line of work, which frames the task as producing an answer together with evidence and evaluating whether the evidence actually entails it, is the closest published formulation of claim-level support checking. Described rather than cited to a specific title, because several papers in that line share closely similar names.

Why the supplied evidence may not be used

• Lost in the Middle: How Language Models Use Long Contexts (Liu et al., 2023), arXiv:2307.03172 — the measured positional effect on whether material in the context is used, which is why an answer can fail while the correct passage was present and ranked well.

Reporting several quantities together

• Holistic Evaluation of Language Models (Liang et al., 2022), arXiv:2211.09110 — the multi-metric argument, applied here to the insistence that support, correctness and sufficiency are reported jointly rather than averaged into one quality figure.

No verified reference is recorded for the four-cell sufficiency-by-correctness reporting used here, for scoring correct abstention as a passing outcome, or for corpus-state pinning across evaluation runs.