Confidence Calibration

What it is

Checking whether the certainty a system expresses corresponds to how often it is right. If a component attaches 0.9 to a thousand outputs, roughly nine hundred of them should be correct; if six hundred are, the number is not a probability and anything built on it is mis-specified. The measurement needs a confidence value per output, an eventual verdict on whether that output was correct, and enough paired observations to compare the two across the range.

Why calibration is not accuracy

A system can be highly accurate and badly calibrated, and badly calibrated in the direction that causes harm: ninety-five percent correct while claiming near-certainty on everything, so the five percent of errors arrive indistinguishable from the rest. It can also be poorly accurate and perfectly calibrated — honestly reporting 0.4 and being right four times in ten — which is a system you can build on, because the low numbers route work elsewhere. Accuracy asks how often the answer is right; calibration asks whether the system knows. Only the second determines whether an abstention threshold, an escalation rule, a selective-answering policy or a confidence shown to a user is doing anything other than adding noise.

Why it matters

Every mechanism that treats some outputs differently from others rests on a score, and placing a threshold on an uncalibrated score is arbitrary. Products also show confidence to users, and a system that hedges at random teaches people to ignore the hedge within a week — after which the signal is pure interface cost and the genuinely uncertain answers look like the rest. The commercially important case is selective answering: a system that answers eighty percent of requests at high reliability and routes the remainder to a person can be worth far more than one that answers everything at middling reliability, and that trade is only constructible if the score separating the two populations means what it says.

Key decisions

• Where the confidence comes from: token-level likelihood, a number the model states, agreement across repeated samples, a trained scorer over the output, or an ensemble. They differ in cost, availability and in what they are actually sensitive to.
• What counts as correct, since calibration is measured against a verdict and a noisy or subjective correctness label puts a ceiling on how well calibration can be assessed at all.
• Which summary is reported. A single expected-error figure hides where the miscalibration is; a reliability curve shows the shape and needs enough observations per bin to be stable.
• Whether the system is recalibrated post hoc against held-out data, and how often that fitting is refreshed as the input distribution moves.
• Whether calibration is reported per segment, because a system well calibrated overall is routinely overconfident on exactly the rare, hard class where the threshold matters.
• What the consumer of the score needs: a threshold for routing, a number shown to a user, or a weight in a downstream aggregation — each tolerates a different kind of error.
• Whether the confidence is recorded on every request alongside the eventual outcome, since this measurement cannot be reconstructed retrospectively from traffic that did not log it.

Failure modes

• Scores from a preference-tuned model are used directly, almost everything sits above 0.9, and the distribution has no resolution left to threshold on.
• Calibration is measured on a balanced evaluation set and applied in production where the base rate is twenty times lower, so the fitted mapping is wrong everywhere.
• A single aggregate calibration error is reported, and the system is well calibrated in the middle and severely overconfident at the top — which is the only region the escalation rule uses.
• Verbalised confidence is collected from the model and treated as a probability, when it is generated text that tracks the phrasing of the question.
• Sample-agreement confidence is high because the model is consistently and confidently wrong, and the mechanism reports certainty on precisely its worst failures.
• The calibrator was fitted once at launch, the input distribution moved, and nothing re-fits or re-measures it.
• Reliability bins contain a dozen observations each, the curve is noise, and a recalibration is fitted to it.
• Confidence is logged and the eventual correctness verdict is not, so the paired data the measurement requires never exists.
• A user-facing uncertainty marker appears on most answers, users stop reading it, and the product keeps paying for a signal nobody acts on.

Related capabilities

• Decision Model Evaluation — where a threshold is placed on the score this capability validates
• LLM-as-a-Judge — a judge that reports its own certainty needs the same treatment as any other scorer
• Online Evaluation — supplies the paired confidence and outcome records this measurement consumes
• RAG Evaluation — whether a system hedges when its evidence was thin is a calibration question
• Human Evaluation — produces the correctness verdicts calibration is measured against

Applies: Production readiness · Design questions