Human Evaluation — References

Sources on measuring agreement between raters, on the known problems with human preference data, and on the obligations that come with asking people to review harmful material.

Agreement

• Cohen's kappa, Fleiss's kappa and Krippendorff's alpha are the standard agreement statistics for, respectively, two raters, many raters with the same category set, and the general case with missing data and varied measurement levels. Named as statistics rather than attributed to particular papers; choosing between them depends on the label type and on whether raters vary per item.
• The literature documenting that annotator disagreement on subjective natural-language tasks is often systematic rather than noise — reflecting genuine differences in interpretation that aggregation destroys — is the strongest argument for reporting the distribution of verdicts rather than their mean. Described rather than attributed to a single title.

Preference judgements and their biases

• Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena (Zheng et al., 2023), arXiv:2306.05685 — cited here for its treatment of human preference collection as the reference standard, and for the position and verbosity effects that apply to human raters as well as to model ones.
• Holistic Evaluation of Language Models (Liang et al., 2022), arXiv:2211.09110 — relevant for its insistence that an evaluation declare its conditions, which for a human campaign means the reviewer population, the guideline and the task form.

Reviewer welfare

• The occupational-health literature on content moderation work, and the industry standards that have grown around it — limited exposure sessions, blurring and greyscale by default, opt-out without penalty, and access to professional support — applies directly to anyone reviewing model outputs in harmful categories. Described rather than cited to a specific publication; the material lives in trust-and-safety practitioner guidance rather than in a canonical paper.

No verified reference is recorded for queue triage by automated scorer confidence, for guideline versioning as an evaluation control, or for rating drift over the course of a long review session in this specific setting.