LLM-as-a-Judge — References

The measured biases of model judges, the evidence on how well their verdicts track human ones, and the work on writing rubrics a model can apply.

The reference study

• Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena (Zheng et al., 2023), arXiv:2306.05685 — the standard reference for model-as-judge agreement with humans, and for position and verbosity bias. The source for the practice of swapping candidate order and for treating agreement with human preference as the measure of a judge's validity.

Rubric-following judges

• The Prometheus line of work on open-weight evaluator models trained to apply a supplied, fine-grained scoring rubric is the clearest published treatment of judging as rubric compliance rather than as general capability. Described rather than cited to a specific title, since the project spans multiple releases with distinct papers.

Why verdicts differ from scores

• Holistic Evaluation of Language Models (Liang et al., 2022), arXiv:2211.09110 — relevant here for its treatment of human and automated judgements as separate instruments reported side by side rather than one standing in for the other.
• Constitutional AI: Harmlessness from AI Feedback (Bai et al., 2022), arXiv:2212.08073 — a model applying an explicit written set of principles to another model's output, which is structurally the same mechanism as a judge applying a rubric and is worth reading for how much weight the written text carries.

Agreement between raters

• Standard inter-rater agreement statistics — Cohen's kappa for two raters, Krippendorff's alpha where raters or categories vary — are the right instruments for reporting judge-against-human agreement, and the reason raw percentage agreement overstates it on skewed label distributions. Named as statistics rather than attributed to a particular paper.

No verified reference is recorded for the effect of rubric decomposition on judge reliability, for score drift across provider model builds, or for self-preference between judge and generator from the same model family. The first two are operational findings; the third is widely observed and, as far as this handbook can confirm, not cleanly isolated in a single published measurement.