Grounding Validation — Patterns

Decompose to atomic claims before judging

Problem. A paragraph judged as a unit returns a single verdict for a mixture of well-supported statements and one invented figure. Whatever the threshold, the verdict is wrong: pass it and the figure ships, fail it and four correct sentences are discarded with it.

Approach. Split the answer into claims small enough that each is independently checkable — roughly, one assertion with its qualifiers attached — and evaluate each against the evidence set. The answer's verdict is then a function of its claims, which allows unsupported claims to be located, removed, or marked individually. Keep the mapping from claim back to the sentence it came from, because the remedy operates on the sentence.

Trade-off. Decomposition is itself a model-driven step with its own error rate, and its characteristic error is dropping a qualifier, which turns a defensible hedged statement into an indefensible bare one. Claim-level checking also multiplies the number of judgements per answer, which is where the cost of this capability actually goes.

Make a correct refusal a passing outcome

Problem. The most valuable behaviour a grounded system has is declining when the sources do not answer the question. A naive validator reads "the available documents do not address this" as a statement unsupported by the documents, which is literally true and exactly backwards.

Approach. Classify the answer's speech act before validating it. Refusals, requests for clarification, statements about the retrieval itself, and procedural text are routed to a different check — one that asks whether the refusal was warranted given the evidence, not whether the evidence entails it. Only assertive content reaches the support test. Include correct refusals in the labelled set used to set thresholds, deliberately over-represented.

Trade-off. Speech-act classification is one more component with one more error rate, and misrouting an assertive answer into the refusal path bypasses the gate entirely. There is also an obvious incentive problem: a generator that learns refusals pass easily has been given a reason to refuse, and nothing in this pattern pushes back on that.

Validate against the admitted context, not the retrieved pool

Problem. Retrieval returns a candidate set; truncation, reranking, deduplication, and token budgets then decide what actually reached the model. If validation runs against the candidate set, an answer can be judged supported by a passage the generator never saw — which means it was supported by coincidence, and the next time the budget cuts differently it will not be.

Approach. Capture the exact context that was sent, as sent, and treat that record as the evidence set. Validate against it and nothing else. The discrepancy between the candidate pool and the admitted context then becomes a visible, measurable quantity, and a rising gap is an early warning that the budget is dropping material the answers depend on.

Trade-off. Retaining the full admitted context for every request is a storage and privacy cost that scales with traffic, and it duplicates content that already exists elsewhere. The stricter check also fails answers that were right, where the model happened to know something the context did not contain, and explaining that failure to a user is genuinely difficult.

Grade the claim types differently

Problem. A uniform threshold treats an unsupported adjective and an unsupported monetary figure as the same quantity of failure. In every domain where this capability matters, they are not: the figure is the entire reason the answer was read.

Approach. Type the claims during decomposition — quantities, dates, named entities, negations, causal statements, qualitative description — and apply a per-type rule. Numeric and named claims fail the answer on a single unsupported instance; qualitative claims contribute to a proportion. Negations and quantifiers get their own treatment, because they are the cases where a topically matching passage is most likely to support the opposite of what was written.

Trade-off. Per-type rules mean per-type thresholds, each needing its own labelled examples, and the typing step adds another classification error to the chain. Strict rules on numeric claims also collide with ordinary rounding and unit conversion, which produce figures no source states literally and which users consider obviously correct.