Hallucination Detection

What it is

Finding assertions a model produced that are not true, in situations where nothing was supplied against which to check them. The defining constraint is the absence of a reference: the output has to be interrogated using the model itself, several samples of it, a second model, or an external authority that can be queried about the specific kind of thing being asserted. Where a designated evidence set does exist, the check against it is a different and much stronger instrument; this one is what remains when there is nothing to compare to.

Why it matters

Fluency is uncorrelated with accuracy. A fabricated statement arrives in the same register, with the same confidence and the same formatting, as everything the system gets right, so a reader has no signal at all short of verifying it personally. In open-ended work — drafting, answering from general knowledge, naming an interface, citing a case — there is no passage to check against and no user with the time to check every sentence. What makes the problem tractable in practice is that one class of fabrication is exactly verifiable: an assertion whose referent either exists or does not. Identifiers, function names, file paths, citations, product codes and addresses can be looked up, and looking them up catches a disproportionate share of what actually harms users.

Key decisions

• Which signal family is used: agreement across independent samples, the model's expressed or token-level uncertainty, a second model's judgement, or lookup of referents in an authoritative system. They catch different things and their costs differ by an order of magnitude.
• Which claims are worth checking at all. Verifiable referents are the subset where a check is exact and cheap, and spending the budget there beats spreading it thinly over prose.
• Cost structure. Sampling-based methods multiply generation cost by the number of samples, which makes them a monitor over a slice of traffic rather than a gate on every request.
• What a positive finding produces: hedging, removing the claim, regenerating, annotating, or escalating. Suppressing the whole answer is almost never right, because most of it was correct.
• Whether findings are shown to users, and what a visible uncertainty marker does to trust once it appears on the majority of answers.
• How the detector is evaluated, given that the ground truth required to evaluate it is the same ground truth whose absence created the problem.
• Whether the thing being checked is only the final answer, or also the intermediate reasoning, the tool arguments, and the plan — each of which can be fabricated independently of the answer.

Failure modes

• Agreement-based methods catch what the model is uncertain about and pass what it is confidently and repeatedly wrong about, which is the larger and more damaging class.
• Token probability is used as the confidence signal on a model whose preference tuning flattened exactly that signal, so the scores separate nothing.
• The detector is validated on short factual answers and deployed on multi-paragraph drafts, where the unit it was scoring never existed.
• Everything is hedged, every answer carries a caution, and users learn to ignore the marker within a week, after which it is pure interface cost.
• An invented citation is verified by asking a model whether it exists, and the prior that produced it confirms it.
• Referents are checked against a stale local mirror, so valid new identifiers are reported as fabrications while withdrawn ones pass.
• The final answer is correct and the reasoning that reached it was invented, and only the answer is checked, so a plausible-looking derivation is reused in a context where it does not hold.
• A detection rate is reported with no false-positive rate beside it, so a detector flagging a third of all answers is recorded as highly effective.
• The detector is run on a sample for cost reasons and the sample is drawn from successful requests only, which excludes the traffic where fabrication is most likely.

Related capabilities

• Grounding Validation — the stronger check available when a defined evidence set exists
• Output Guardrails — the checkpoint where a hedge, an annotation, or a suppression is applied
• Tool Guardrails — catches invented identifiers at the point they would be acted on
• Human Approval Gates — where an answer too important to ship unverified is sent
• Content Moderation — the other judgement over generated text, aimed at harm rather than truth

Applies: Production readiness · Design questions