Regression Evaluation — References

Material on separating a real difference from run-to-run variation, and on comparison practices borrowed from performance engineering.

Variance and significance

• The machine-learning reproducibility literature arguing that single-run results are not reportable, and that a spread across seeds or repeats must accompany any claimed improvement, is the direct argument for a published noise floor. Described as a body of work rather than attributed to one title, since the point is made repeatedly across several venues.
• Bootstrap resampling over per-item scores is the standard non-parametric way to put an interval around an evaluation aggregate without assuming a distribution, and paired comparison over the same items is substantially more sensitive than comparing two independent aggregates. Named as statistical methods rather than attributed to a specific paper.

Multiple comparisons

• Reporting many segments and metrics per run means many opportunities for a spurious difference, and the standard corrections for multiple comparisons apply directly. Named as a statistical concern; no single reference is recorded.

Drift in the instrument

• Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena (Zheng et al., 2023), arXiv:2306.05685 — cited here because the judge biases it documents are also sources of run-to-run movement, which is why a judge-scored suite has a wider noise floor than a deterministically scored one.

Continuous comparison practice

• Continuous-benchmarking practice from performance engineering — a fixed workload, a maintained baseline, automatic detection of step changes and slow trends, and an explicit noise threshold — transfers almost unchanged to evaluation suites, with the caveat that the variance here is far larger. Described as a practice; the material lives in tooling documentation rather than in a canonical publication.

No verified reference is recorded for instrument-identity gating of comparisons, for the dual-baseline arrangement described here, or for flake classification in model evaluation suites.