Evaluation and Quality Engineering
Decision Model Evaluation — Stack
Decision model evaluation › Stack
Technology landscape as of 2026-10-10. Knowledge cutoff May 2026.
Selection criteria
The computation is well-understood and decades old; what differs between options is whether they support the operating-point reasoning a production decision needs, and whether they can watch that operating point over time.
- Per-class and per-segment reporting. Whether breakdowns are first-class or something you assemble by hand after every run.
- Threshold analysis. Whether the tool plots the metric against the threshold and supports choosing an operating point under a stated cost ratio, rather than reporting one number at the default cut.
- Base-rate honesty. Whether the tool makes the prevalence of each class visible in its output, since every precision figure is a statement about a prevalence.
- Confidence-interval support. Whether per-class figures come with uncertainty, because a recall of 0.8 on forty examples is not a measurement.
- Drift monitoring. Whether the same definitions can run continuously over production predictions, including against labels that only arrive weeks later.
- Calibration output. Whether reliability curves are produced alongside discrimination metrics, since the threshold is placed on the score and not on the ranking.
- Group fairness slices. Whether differential error rates across defined groups can be reported as part of the standard run rather than as a special project.
Options
| Option | Best for | Watch out for |
|---|---|---|
| Classical metric libraries (scikit-learn's metrics module and equivalents in other languages) | Everything about the core computation, available immediately and with sensible defaults | Defaults assume balanced data and a 0.5 cut; macro and micro averaging differ in ways that quietly change conclusions |
| Model-evaluation and drift platforms (Evidently, Deepchecks and comparable open-source projects) | Standing reports over production predictions with distribution and drift checks included | Built around tabular features; a decision produced by a language model has no feature vector to monitor |
| Experiment trackers with metric comparison (MLflow, Weights & Biases and similar) | Keeping threshold choices, per-class results and the data version attached to a specific model run | Comparison is across runs rather than across operating points; the cost-ratio reasoning is not represented |
| Calibration-specific tooling (reliability diagrams, isotonic and Platt scaling implementations) | Making a score usable as a probability before a threshold is placed on it | Fitting a calibrator needs held-out data and re-fitting whenever the population shifts |
| Business-intelligence dashboards over a predictions table | Segment breakdowns the business already defines, and outcomes that arrive in the warehouse later | No statistical machinery; confidence intervals and significance are whatever the analyst remembered |
| Annotation tooling for building and auditing the label set (Label Studio, Argilla and comparable) | Problems where labels are human decisions and their disagreement rate needs measuring | Label quality work is a standing programme, not a setup task |
| Fairness and slice-analysis toolkits (Fairlearn, slice-discovery libraries and comparable) | Finding the segment where the error rate is double, including ones nobody thought to define | Slice discovery over many candidate cuts finds spurious ones; every finding needs a prevalence and an interval beside it |