Evaluation and Quality Engineering
RAG Evaluation — Stack
Rag evaluation › Stack
Technology landscape as of 2026-10-10. Knowledge cutoff May 2026.
Selection criteria
The components here take a question, an answer, and the passages the model was shown, and return judgements about support, correctness and use of evidence. What separates them is granularity, what evidence they insist on, and whether they can tell you where in the pipeline a problem is.
- Context fidelity. Whether the tool evaluates against the exact assembled context or against whatever the retriever returned, which are different systems once truncation exists.
- Claim-level output. Whether an unsupported fragment is localised to a sentence, or the answer gets one score.
- Abstention handling. Whether a correct refusal is representable as a pass, or is structurally scored as unsupported.
- Reference-answer independence. Whether the scorers work without a written reference, since most teams have far more questions than adjudicated answers.
- Joint reporting. Whether retrieval sufficiency and answer quality can be crossed into one table rather than reported as two averages.
- Judge control. Whether the underlying judge model, its build and its prompt are visible, pinnable and replaceable.
- Corpus pinning. Whether a run can be bound to a stated corpus state, so two comparisons are not silently taken against different documents.
- Cost per item. Claim decomposition plus per-claim judging is several model calls per answer, which sets the practical size of the suite.
Options
| Option | Best for | Watch out for |
|---|---|---|
| Purpose-built retrieval-augmented evaluation libraries (Ragas, TruLens, DeepEval's grounding and relevance metrics and comparable) | Getting faithfulness, answer relevance and context-use metrics running in an afternoon | Metric definitions differ between libraries under similar names; defaults become thresholds; most judge the retrieved set rather than the assembled context |
| Natural-language-inference entailment models applied per claim | Cheap, fast, reproducible support checking once the answer is decomposed into claims | Trained on short sentence pairs and degrade on long passages and on claims needing multi-passage synthesis |
| Trained faithfulness and summary-consistency scorers (hallucination-detection models published by vendors and research groups) | A single calibrated support number per answer at low cost and high reproducibility | Usually trained for summary-against-source; behaviour on retrieval-augmented answers with many passages needs validating on your own data |
| Hand-written judge prompts over the assembled context | Products whose notion of support is domain-specific, and where the evidence set must be the exact context | Everything is yours to build: decomposition, abstention routing, ordering control, agreement measurement |
| Evaluation platforms with trace-level inspection (LangSmith, Langfuse, Arize Phoenix, Braintrust and similar) | Seeing the retrieved passages, the assembled prompt and the answer together while investigating | Scoring semantics are the platform's; the joint retrieval-and-answer view often has to be assembled downstream |
| Citation resolution and span-alignment checkers | Verifying markers point at supplied passages and locating the span a claim rests on, deterministically and almost free | Resolution is not support; a resolvable marker on an unsupported sentence passes every structural check |
| Synthetic question generation from the corpus (question generators seeded by passages) | Producing broad coverage quickly when no curated question set exists | Questions generated from a passage are answerable from that passage by construction, so the suite under-represents the multi-hop and no-answer cases that actually fail |