Decision Model Evaluation

What it is

Measuring a component whose output is a choice from a known, finite set: a category, a route, an approve-or-decline, a priority band, a yes to escalating. Because the output can be compared exactly against a recorded correct choice, the classical apparatus applies without adaptation — a confusion matrix, precision and recall per class, a threshold on a score, and an explicit decision about which kind of mistake is worse. Nothing here concerns the wording of generated text; the thing being judged either picked the right option or did not.

Why it matters

These components sit at the front of systems and decide what happens to a request, so their errors are structural rather than cosmetic. A misrouted request is answered by the wrong specialist; a wrongly approved transaction is a loss; a wrongly declined one is a customer who leaves. They are also the places where an aggregate accuracy figure is most dangerously reassuring: a classifier that never fires on the class that occurs two percent of the time reports ninety-eight percent accuracy and is useless. And where the choice is produced by a language model rather than a trained classifier, the same arithmetic still governs, but the score the threshold is set against is far less trustworthy.

Key decisions

• Which error costs more, stated as a ratio before any threshold is chosen. Without it, the operating point is set by whichever metric the tooling defaults to.
• Where the operating threshold sits, and whether it is set per class, per tenant or per segment. A single global threshold on a population with different base rates is several wrong thresholds at once.
• How the labels were produced. If the ground truth is a human decision — an approval, a triage category — then it carries that human's error rate, and the measured ceiling is agreement with people rather than correctness.
• Which figure is reported. Accuracy on skewed data says almost nothing; precision and recall per class say what happens to each kind of request; a threshold-independent summary says how good the ranking is and nothing about the operating point actually in use.
• Whether the evaluation set preserves the production base rate or is rebalanced. Rebalancing makes a model comparable and makes its precision figure fictional.
• Whether an abstention is permitted as an outcome, and how it is scored, since a model that declines to decide on its hardest ten percent may be far more valuable than one that guesses.
• How the measurement is segmented across groups where a differential error rate would be a fairness or compliance problem rather than a quality one.

Failure modes

• Accuracy is the headline on a problem where one class is three percent of traffic, and a model that never predicts that class is recorded as excellent.
• The threshold was tuned on the same data used to report the final numbers, so the reported precision is the best of several hundred thresholds and will not reproduce.
• A threshold-free summary statistic is used to pick between two models, and the one that wins overall is worse at the only operating point the product can afford to run.
• Labels come from past human decisions that were themselves systematically wrong in one direction, and the model is scored as correct precisely when it reproduces that bias.
• The class balance shifts in production after a marketing campaign, the threshold stays where it was, and precision collapses without any model change.
• A model is evaluated on a rebalanced set where the rare class is a third of the data, and the precision number is quoted to stakeholders as if it described production.
• Confidence scores from a language model asked to classify are treated as probabilities, and the threshold is placed on a number that was never calibrated against anything.
• The decision is evaluated in isolation, and nobody measures what the downstream system does with a wrong decision, so a recoverable misroute and an irreversible decline are weighted identically.
• Per-class results are strong and the error rate for one customer segment is double the rest, because the report was never broken down that way.

Related capabilities

• Confidence Calibration — whether the score a threshold is placed on means what it appears to mean
• Deterministic Evaluation — the comparison mechanism that makes exact scoring possible here
• Golden Datasets — where the labelled cases and their base rates are curated
• Regression Evaluation — detecting that a decision boundary has moved between releases
• Evaluation Gates — where a per-class floor becomes a release condition

Applies: Production readiness · Design questions