Confidence Calibration — References

The core result on modern models being overconfident, the methods for correcting it, and the evidence on whether a model's own statements about its certainty are worth anything.

The core result and the standard corrections

• On Calibration of Modern Neural Networks — Chuan Guo, Geoff Pleiss, Yu Sun and Kilian Q. Weinberger, International Conference on Machine Learning, 2017. The demonstration that modern networks are systematically overconfident, the introduction of temperature scaling as a ranking-preserving correction, and the reference for reliability diagrams and expected calibration error as reporting instruments. Cited by name.

Whether a model knows what it knows

• Language Models (Mostly) Know What They Know — Saurav Kadavath and colleagues at Anthropic,
  2022. Evidence that models can be trained to predict whether their own answers are correct, together with the limits of that self-assessment as tasks move away from the calibration distribution. Cited by name.
• Teaching Models to Express Their Uncertainty in Words — Stephanie Lin, Jacob Hilton and Owain Evans, Transactions on Machine Learning Research, 2022. Verbalised confidence as a trainable behaviour, and the comparison against log-probability-derived confidence that is the basis for choosing between the two sources. Cited by name.

Consistency as a confidence signal

• SelfCheckGPT: Zero-Resource Black-Box Hallucination Detection for Generative Large Language Models — Potsawee Manakul, Adian Liusie and Mark J. F. Gales, Conference on Empirical Methods in Natural Language Processing, 2023. Sampling several responses and measuring their mutual consistency, with no model internals required. Relevant here as the main confidence source available when the endpoint exposes no likelihoods, and as the clearest statement of that method's assumption. Cited by name.

Guarantees rather than point estimates

• Conformal prediction — the framework producing prediction sets with a marginal coverage guarantee at a chosen error rate, given exchangeable data — is the principled alternative to a calibrated point confidence, and its assumptions are exactly what a shifting input distribution violates. Named as a framework; the material is covered in textbook-length treatments rather than in one canonical paper.

No verified reference is recorded for the degradation of likelihood calibration under preference tuning as a measured effect, for per-segment calibration reporting in language-model products, or for the effect of user-facing uncertainty markers on sustained trust.