Regression Evaluation

What it is

Comparing two runs and deciding whether the difference between them is real. The inputs are two sets of results — a baseline and a candidate — and the output is a judgement: this moved, by this much, on these items, for this reason. It is the analysis layer rather than the measurement layer, and its central problem is that every evaluation has variance, so a difference always exists and most differences mean nothing. Deciding that a movement is real is this capability's job; deciding what to do about it is somebody else's.

Why it matters

Quality in these systems degrades through accumulation rather than through single bad releases. No individual prompt edit is obviously wrong, and twenty of them over a quarter produce a product that is measurably worse than it was, with no commit anyone can point at. Comparison against a maintained baseline is the only mechanism that notices, and it is the only one that can attribute a movement to a change while the change is still fresh enough to revert cheaply. The second reason is more mundane: without a principled account of noise, teams investigate differences that are not there, and the credibility of the whole evaluation programme is spent on false alarms.

Key decisions

• What the baseline is: the previous release, a fixed long-lived reference configuration, or a rolling window of recent runs. Each gives a different meaning to the word regression.
• How run-to-run variance is estimated, since no threshold on a difference is defensible without it, and the only honest way to get it is repeated runs of an unchanged system.
• Whether comparison is on the aggregate or per item. Aggregates hide compensating movement; per item comparison finds everything, including hundreds of changes nobody cares about.
• What counts as material. A statistically detectable difference and a difference worth blocking a release over are different thresholds, and conflating them makes every comparison alarming.
• How scoring variance is separated from system variance, because a nondeterministic judge produces differences all by itself.
• How often the baseline is refreshed, and what is lost when it is. A baseline that advances with every release cannot see slow decline; one that never advances eventually compares against a system nobody remembers.
• Whether the comparison is attributable: if several changes shipped between the two runs, the analysis can identify a movement and not its cause.

Failure modes

• The aggregate is flat and sixty items improved while sixty degraded, so a change that redistributed quality across the product is recorded as having done nothing.
• No variance baseline exists, a two-point movement is investigated for a week, and the eventual finding is that the suite moves two points on its own.
• Every per-item change is reported, the diff runs to four hundred lines, and within a month nobody opens it.
• The baseline is always the previous release, each release is within noise of the last, and the system loses fifteen points over a year without a single alert.
• Scores from a judge whose provider updated the model are compared against scores from before the update, and the instrument change is reported as a product regression.
• A regression is detected after eleven changes shipped together, and the investigation is a bisection nobody budgeted for.
• The comparison is run only against the suite's own aggregate, so a regression confined to one tenant or one language never reaches the threshold.
• Flaky items fail intermittently, get marked as known failures to quiet the report, and permanently stop contributing signal.
• A candidate scores better because an item's expected answer was corrected in the same commit, and the improvement is attributed to the model change.

Related capabilities

• Evaluation Gates — the release control that acts on a movement this analysis confirms
• Offline Evaluation — produces the two runs being compared
• Evaluation Dataset Versioning — establishes that both runs were scored against the same items
• Online Evaluation — where a movement too subtle for a fixed suite eventually shows up
• Decision Model Evaluation — the per-class view that reveals a regression an aggregate hides

Applies: Production readiness · Design questions