Regression Evaluation — Patterns

Establish the noise floor before interpreting any difference

Problem. A candidate scores 0.81 against a baseline of 0.83 and an investigation starts. It ends days later with the discovery that running the baseline twice also produces 0.81 and 0.83, because sampling, judge variance and retry behaviour move the number on their own.

Approach. Run the unchanged baseline configuration repeatedly — several times, on a schedule, with nothing altered — and record the distribution of the aggregate and of per-item verdicts. Publish the resulting spread as a property of the suite, and require any reported movement to be compared against it. Recompute it whenever the suite, the scorers or the model configuration changes, since all three affect it.

Trade-off. The repeated runs cost real money and produce no new coverage, and the noise floor dates quickly. A published spread also sets a floor on detectable effects, which makes it official that the suite cannot see small regressions — true, unwelcome, and frequently argued with.

Diff per item, in both directions

Problem. Aggregate comparison is blind to redistribution. A change that fixes sixty items and breaks sixty others leaves the mean untouched, and that is not a neutral release — it is the most dangerous kind, because the broken sixty are now a surprise in production.

Approach. Join the two runs by stable item identifier and classify every item into improved, degraded, unchanged, or newly erroring. Report the four counts before reporting the aggregate, and surface degraded items with their before-and-after outputs. Keep the categories visible even when the aggregate is flat, since that is exactly the case the pattern exists for.

Trade-off. Per-item diffs on a suite of any size produce more changed items than anyone will review, so the list needs its own prioritisation — by severity stratum, by magnitude, by whether the item was previously passing — and that prioritisation is a second place for a real regression to get filtered out.

Hold the instrument fixed, or declare that it changed

Problem. Scores are compared across a boundary where the scorer itself changed: a judge model updated by its provider, a rubric improved, a metric's implementation fixed. Every number moves, and the movement is indistinguishable from a product regression.

Approach. Record the full instrument identity on every result — scorer versions, judge model build, rubric revision, dataset version — and make the comparison tool refuse to diff across a difference in any of them. When an instrument must change, re-score the stored baseline outputs with the new instrument, publish the offset between old and new on identical data, and restart the comparison series from the re-scored baseline.

Trade-off. Re-scoring a stored baseline requires that the baseline's raw outputs were kept, which is a storage commitment made long before anyone knew it would be needed. The discipline also makes improving a scorer costly, which quietly discourages it.

Keep a long-lived reference alongside the moving baseline

Problem. Comparing each release to the previous one catches step changes and is structurally blind to drift. Twenty consecutive releases, each within noise of the last, can add up to a large decline that no single comparison ever flagged.

Approach. Maintain two baselines. The moving one is the previous release and answers "did this change break anything". The fixed one is a configuration deliberately frozen months ago, re-run on the same cadence, and answers "where are we compared to where we were". Report both. Advance the fixed reference rarely, deliberately, and with an explicit record of the offset at the moment it moves.

Trade-off. Two baselines mean double the runs and a standing requirement to keep an old configuration executable, which decays as dependencies, prompts and model availability change. Eventually the frozen reference cannot be run at all, and the series it anchored ends.