Decision Model Evaluation — References

The arithmetic of operating points under skewed data, the evidence that model scores are not probabilities, and the reporting discipline that keeps a per-class result honest.

Why the base rate dominates

• The Base-Rate Fallacy and its Implications for the Difficulty of Intrusion Detection — Stefan Axelsson, ACM Conference on Computer and Communications Security, 1999. The result that when the event of interest is rare, the limiting factor on a usable system is the false alarm rate rather than the detection rate. The single most useful piece of arithmetic for anyone placing a threshold on production traffic. Cited by name.

Curves and what they hide

• The comparison between precision–recall curves and receiver operating characteristic curves on highly imbalanced data — the result that the latter can look strong while the operating region the product needs is unusable — is standard and well established in the machine-learning literature. Described rather than attributed to one title, because several papers in that line are commonly cited interchangeably.

Scores are not probabilities

• On Calibration of Modern Neural Networks — Chuan Guo, Geoff Pleiss, Yu Sun and Kilian Q. Weinberger, International Conference on Machine Learning, 2017. Evidence that a classifier's reported scores are systematically overconfident, which is why a threshold has to be derived from your own data rather than read off the score as a probability. Cited by name.

Reporting

• Model Cards for Model Reporting — Margaret Mitchell, Simone Wu, Andrew Zaldivar and colleagues, Conference on Fairness, Accountability, and Transparency, 2019. The argument for declaring evaluation conditions and reporting disaggregated results across groups, which is the discipline that catches a doubled error rate in one segment. Cited by name.

No verified reference is recorded for cost-ratio elicitation in practice, for deferral rates as a jointly optimised quantity against human queue capacity, or for threshold selection on scores emitted by a general-purpose language model asked to classify.