Human Evaluation — Patterns

Measure agreement before believing any aggregate

Problem. A campaign returns a mean rating of 3.8 and everyone treats it as a measurement of the system. If the reviewers disagree with each other more than the candidates differ, the number describes the reviewer pool and the guideline, not the product.

Approach. Overlap a deliberate fraction of items across reviewers from the start, compute an agreement statistic appropriate to the label type, and publish it on the face of every result. Set a minimum agreement below which results are not reported as findings but as evidence that the task is badly specified. When agreement is low, fix the guideline or the question — adding reviewers to a poorly posed question buys precision around the wrong value.

Trade-off. Overlapping items costs reviewer capacity that produces no new coverage, and the fraction needed for a stable agreement figure is larger than teams want to spend. High agreement is also not validity: reviewers who have all absorbed the same wrong reading of the guideline agree perfectly.

Blind the reviewer and randomise the order

Problem. Reviewers shown which system produced which output rate the one they expect to be better as better, and reviewers shown two candidates in a fixed order develop a positional habit. Both effects are large enough to determine the result of a close comparison.

Approach. Strip every identifying cue from the presented output — model name, formatting conventions that give it away, internal identifiers — and randomise candidate position per item, recording the true assignment separately. Analyse for position effects as a standing check; a rising rate of position-correlated verdicts means reviewers have stopped reading.

Trade-off. Thorough blinding is harder than it looks, because candidates often differ in characteristic style, and a reviewer who recognises the style is unblinded regardless of what the interface shows. Blinding also removes context that would legitimately inform judgement in some domains, which has to be weighed rather than assumed away.

Triage the queue so expensive attention reaches hard cases

Problem. Expert review time is the scarcest resource in the stack, and a uniformly sampled queue spends most of it confirming that routine answers are fine. The cases where expert judgement would change a decision are a small fraction of what the expert actually sees.

Approach. Run cheap scorers first and route by their output: items an automated judge scores confidently at either extreme go to a light confirmation pass or none at all, while items in the uncertain band, items where two scorers disagree, and items from high-severity strata go to experts. Keep an unrouted random sample flowing to experts as well, sized to detect the failures the routing itself is blind to.

Trade-off. Routing by an automated scorer means the human sample is shaped by that scorer's biases, and the cases it is systematically wrong about are exactly the ones it will route away. The random control sample is what limits that damage, and it is the first thing cut when capacity is short.

Treat the guideline as a versioned artefact with worked examples

Problem. Review instructions live in a document that gets edited whenever a reviewer asks a question. Verdicts from before and after the edit are pooled, campaigns months apart are compared as though they measured the same thing, and no record exists of what the question was at any given time.

Approach. Version the guideline, record its revision on every verdict, and build it around worked examples rather than adjectives — several reviewed items per scale point, including the hard cases that generated the clarification. Changes go through the same review as code, and a material change starts a new campaign rather than continuing the old one. Re-score a small calibration set after every change to measure the offset the change introduced.

Trade-off. Versioning discourages improving the guideline, which is usually the cheapest way to improve data quality, and worked examples anchor reviewers so strongly that items unlike them get judged by the nearest example rather than by the criterion.