RAG Evaluation — Patterns

Score against the assembled context, claim by claim

Problem. An answer is a mixture. Four sentences restate the sources accurately, one adds a figure that appears nowhere, and a whole-answer verdict has to choose between passing all five and failing all five. Meanwhile the support check often runs against the candidate pool rather than the passages that fit in the window, so the invented figure may be credited to a passage the model never saw.

Approach. Persist the exact context as sent, with passage identifiers, as part of the trace. Split the answer into independently checkable assertions with their qualifiers attached, and judge each one against that context alone, recording which passage supports it. The answer's result becomes a vector of claim verdicts, which both localises the unsupported fragment and produces the evidence for a citation check.

Trade-off. Decomposition is a model step with its own error rate, and the error it makes most is dropping a qualifier, turning a defensible hedge into an indefensible assertion. The cost is several judgements per answer, which caps suite size; and a claim can be individually supported while the answer as a whole is misleading through selection.

Cross retrieval sufficiency with answer quality

Problem. Reporting an average support score and an average retrieval score side by side hides the only distinction that drives action. A system where the evidence was present and the answer was still wrong needs generation work; one where the answer was right despite absent evidence is running on model memory and will fail silently as questions move.

Approach. For each item, record two facts: whether the supplied context actually contained what was needed, and whether the answer was correct. Report the four-cell table rather than two margins, with the counts for each cell tracked release over release. The cell where evidence was sufficient and the answer wrong is the generation backlog; the cell where evidence was insufficient and the answer right is the measurement of how much the system is not using retrieval at all.

Trade-off. It requires a sufficiency judgement per item, which is extra annotation and is itself contestable — sufficiency depends on what counts as the required information. The table is also harder to put in a release summary than a single percentage, so it tends to be built once and then quietly replaced by an average.

Make the no-answer cases a first-class stratum

Problem. Suites are built from questions the corpus answers, because those are the questions somebody wrote reference answers for. In production a large share of questions are not covered at all, and the behaviour on them — abstain, or synthesise from the nearest passage — is both the most consequential and the least measured.

Approach. Deliberately include questions with no supporting evidence: topics outside the corpus, questions about a document that was deleted, questions whose answer the corpus contradicts. Define the correct behaviour for each as an explicit abstention or a stated limitation, and route these items to a check that asks whether the refusal was warranted, not whether the sources entail the response. Report this stratum separately and never average it into the headline.

Trade-off. Rewarding abstention creates an incentive to abstain, and a system tuned on a suite with a heavy no-answer stratum learns to hedge on answerable questions. The two strata therefore have to be reported together and watched against each other, which is one more thing that gets dropped when the dashboard is simplified.

Pin the corpus state to the run

Problem. Two evaluations a week apart are compared. In between, a hundred documents were updated, one source was re-crawled, and the answer to nine of the questions legitimately changed. The score moved and the system did not.

Approach. Bind every run to an explicit corpus state — a snapshot identifier, an index version, or a content digest over the retrievable set — and record it with the results. Refuse to compare two runs taken against different states without an explicit acknowledgement, and when the corpus must advance, re-run the previous configuration against the new state to measure the offset before attributing anything to a code change.

Trade-off. Pinning means evaluating against a corpus that is not the one production serves, so the suite becomes progressively less representative exactly as it becomes more comparable. Maintaining snapshots of a large corpus is also real storage and real operational work, and the re-baselining run doubles the cost of every corpus advance.