Model Benchmarking — References

Harnesses, benchmark designs, and the methodological problems a model comparison has to survive.

Harnesses

• Inspect AI — an evaluation framework from the UK AI Safety Institute, with explicit solver, scorer, and log abstractions.
• LM Evaluation Harness — EleutherAI; the de facto implementation of many standard academic benchmarks.
• OpenAI Evals — a registry-based format for defining and running evaluation tasks.
• Stanford CRFM HELM — holistic evaluation reporting accuracy, calibration, robustness, efficiency, and cost as parallel metrics.

Benchmark designs worth reading before writing your own

• Measuring Massive Multitask Language Understanding (Hendrycks et al., 2020), arXiv:2009.03300 — the MMLU construction, and a clear example of what a broad academic average does and does not tell you.
• Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena (Zheng et al., 2023), arXiv:2306.05685 — the standard reference for model-as-judge agreement with humans, and for position and verbosity bias.
• Holistic Evaluation of Language Models (Liang et al., 2022), arXiv:2211.09110 — the multi-metric argument that quality alone is an inadequate basis for a model decision.

Contamination and measurement validity

• Documenting Large Webtext Corpora: A Case Study on the Colossal Clean Crawled Corpus (Dodge et al., 2021), arXiv:2104.08758 — benchmark contamination found in a widely used pretraining corpus.
• Chatbot Arena / LMArena — pairwise human preference rankings; a useful shortlisting signal and not a substitute for measurement on your own tasks.

Load measurement

• k6 documentation — scripted load generation with controlled arrival rates and percentile reporting, for the latency half of a comparison.

No reference is recorded for statistical methodology specific to language-model comparisons; standard paired-comparison and bootstrap confidence-interval methods apply, and no authoritative source specific to this setting is known.