RAG Evaluation

What it is

Judging the answer a retrieval-augmented system produced, given the evidence it was actually shown. Three questions have to be separated and all three asked: is every assertion in the answer supported by the supplied passages, is the answer correct, and did it use what it was given. They come apart in practice — an answer can be fully supported and wrong because the passage was wrong, or unsupported and right because the model already knew — and a measurement that collapses them into one score cannot tell you which system you have.

Why this is not retrieval measurement

Whether the right passages were found is answerable without generating anything, is cheap enough to run on every index change, and localises a defect to a specific stage of the pipeline. That measurement is a different discipline with its own judged sets and its own depth-dependent metrics, and it is the right instrument for tuning segmentation, fusion and ranking. What it cannot do is tell you what the system said. This capability starts where the context window has been assembled and asks what happened next: whether the generator used the evidence, ignored it, contradicted it, or filled a gap from its own weights. The two measurements are complementary and neither substitutes for the other — strong retrieval with a generator that paraphrases badly, and weak retrieval with a generator that abstains correctly, are both common and are indistinguishable from either number alone.

Why it matters

The defining risk of a retrieval-augmented product is an answer that reads as sourced and is not. Users grant these systems more trust than a bare model precisely because the architecture implies evidence, and the failure that exploits that trust — a fluent synthesis with a citation attached to a passage that says something else — is invisible to anyone not reading the sources. It is also the measurement that settles the recurring argument about where a defect lives, because the joint view across retrieval and generation is the only thing that shows a regression was caused by a prompt change rather than by the index everyone suspected.

Key decisions

• What the evidence set is when support is judged: the full candidate pool, or the exact context the model received after truncation and reordering. Only the second asks a question about the system that ran.
• What unit support is assessed at. A whole answer gets one verdict for a mixture of supported and invented content; individual assertions localise the defect and multiply the cost of judging an item.
• How a correct abstention is scored, since the most valuable behaviour a grounded system has is declining when the sources do not answer, and a naive support check reads that as unsupported.
• Whether correctness is judged against a reference answer or against the passages. The first catches a wrong source; the second catches an unfaithful generator; they disagree in exactly the cases that matter.
• Whether evaluation reports the joint distribution over retrieval sufficiency and answer quality, rather than two independent averages that cannot be crossed.
• How citations are checked — that the marker resolves to a supplied passage, and separately that the passage supports the sentence it is attached to.
• Whether the corpus is pinned for the duration of a comparison, given that a document edited overnight changes both what is retrievable and what counts as a correct answer.

Failure modes

• Support is judged against everything retrieval returned, while the model only ever saw the first four passages, so answers are credited as grounded in evidence that was truncated away.
• An answer is correct because the model knew the fact independently, the retrieved passages were irrelevant, and the system is recorded as working — until the question shifts to something it does not know.
• Every assertion is supported by the sources and the sources are out of date, so the measurement certifies a confidently wrong answer as grounded.
• The suite rewards answering, so a system that correctly says the documents do not cover this scores below one that confabulates from the nearest passage.
• Citation checking verifies only that markers resolve to real passages, and the model learns to attach plausible markers to sentences those passages do not support.
• Answer quality and retrieval quality are each averaged and reported separately, and the category that matters most — evidence was present and the answer was still wrong — is never counted because it requires crossing the two.
• The evaluation runs against a fixture corpus that has been complete and current since the day it was built, while production answers from an index with a month of lag.
• Items are scored by a judge shown the answer and the passages but not the question, so it rates internal consistency rather than whether anything was answered.
• A long answer with one invented clause scores well because the support metric is a proportion, and the invented clause is the number the reader acted on.

Related capabilities

• LLM-as-a-Judge — the mechanism that assesses support and correctness here
• Golden Datasets — supplies the questions, reference answers and required assertions
• Deterministic Evaluation — checks citation markers resolve before anything semantic is asked
• Task Completion Evaluation — the broader question of whether the user's goal was met
• Confidence Calibration — whether the system's hedging tracks when its evidence was thin

Applies: Production readiness · Design questions