Evaluation and Quality Engineering
Regression Evaluation — Stack
Regression evaluation › Stack
Technology landscape as of 2026-10-10. Knowledge cutoff May 2026.
Selection criteria
The components store results, align them, and present a difference. What separates them is whether they can tell a real movement from noise and whether they make a diff that a human will actually read.
- Item-level alignment. Whether two runs can be joined per item by a stable identifier, which is the precondition for any diff worth having.
- Variance estimation. Whether repeated runs of an unchanged configuration can be stored and summarised, so that a comparison has a reference for ordinary movement.
- Instrument-change detection. Whether the tool notices that the scorer, judge build or dataset version differs between the two runs and refuses to compare silently.
- Segmented comparison. Whether differences can be computed per stratum — tenant, language, intent class, severity — rather than only over the whole suite.
- Diff presentation. Whether changed items are surfaced with before-and-after outputs side by side, which is the only form in which a diff gets read.
- Baseline management. Whether a baseline can be pinned, promoted and annotated, rather than implicitly being whatever ran last.
- History depth. Whether enough runs are retained to see a slow slide, since year-long drift is invisible in a two-run comparison.
- Flake accounting. Whether intermittently failing items are identified as such rather than being silently excluded or permanently muted.
Options
| Option | Best for | Watch out for |
|---|---|---|
| Experiment trackers with run comparison (MLflow, Weights & Biases, Neptune and comparable) | Keeping runs, configurations and metrics together with a built-in comparison view | Comparison is metric-first; per-item diffs and variance baselines usually have to be modelled by hand |
| Evaluation platforms with built-in run diffing (LangSmith, Langfuse, Braintrust and similar) | Side-by-side output comparison for changed items, which is where the diagnosis actually happens | Statistical treatment of noise is typically thin; a difference is shown without any sense of whether it is real |
| Snapshot and approval testing tooling (approval-test libraries, golden-file frameworks) | Deterministic outputs where any textual change should be reviewed deliberately | Generative outputs change constantly for harmless reasons; the review burden becomes unsustainable and approvals get rubber-stamped |
| Statistical testing in analysis code (bootstrap resampling, paired tests over per-item scores) | Honest statements about whether a difference exceeds the suite's own variance | Requires per-item scores and someone who will interpret the result correctly; paired designs are easy to apply to unpaired data by mistake |
| Warehouse tables with scheduled comparison queries | Long history, slow-drift detection, and segmentation by any dimension already modelled | No interactivity during an investigation; the diff is a query result, not a readable comparison |
| Continuous-benchmark dashboards borrowed from performance engineering | Tracking a metric over many runs with automatic detection of step changes and slow trends | Designed for low-variance measurements; language-model suites are noisier than their default thresholds assume |
| Purpose-built diff tooling over stored run artefacts | Teams whose per-item artefacts are rich and whose comparison needs are specific | Somebody owns it; alignment, flake detection and presentation are each more work than they look |