LLM-as-a-Judge — Patterns

Validate against human labels before trusting any verdict

Problem. A judge returns a confident number for every item, and the number looks like a measurement. Nothing in the output indicates whether the judge understood the rubric, whether it agrees with the people whose opinion the rubric was meant to encode, or whether it agrees with itself.

Approach. Draw a sample that spans the full quality range rather than the typical case, have people label it independently against the same written rubric, and report the judge's agreement with them as a first-class metric published alongside every score. Measure agreement separately for clear passes, clear failures and the ambiguous middle, because an overall figure is carried by the easy cases. Re-run the comparison whenever the rubric, the judge build or the output distribution changes, and treat a drop in agreement as an evaluation incident.

Trade-off. It converts a cheap automated metric into one that carries an ongoing human cost, and the sample has to be large enough for the agreement figure to mean anything — which, for the ambiguous middle, is larger than teams expect. There is also no escape from the circularity: agreement is measured against human labels that themselves disagree.

Compare two candidates rather than scoring one

Problem. Absolute scores from a model drift. The same response scored this month and next month lands differently, scores bunch around the middle of any scale whose points are not anchored, and a half-point movement across a suite is unexplainable.

Approach. Present the judge with the input and two responses, ask which better satisfies the rubric, and permit an explicit tie. Run each pair in both orders and discard or average the cases where the verdict follows the position rather than the content. Aggregate the pairwise outcomes into a win rate against a fixed reference configuration, so the result stays interpretable across releases.

Trade-off. A win rate says one candidate is better, never how good either is, so it cannot support a release threshold on its own. Running both orders doubles the cost, and the fraction of order-dependent verdicts is itself a number somebody has to watch, because when it rises the instrument has stopped working.

Decompose the rubric into independent binary questions

Problem. "Rate the overall quality from one to five" produces a single impression that correlates with fluency and length more than with any property anybody cares about, and when it moves, nothing explains why.

Approach. Break the rubric into specific yes-or-no questions, each about one checkable property — did it answer the question asked, did it state the required figure, did it stay inside the supported scope, did it avoid the forbidden recommendation. Ask them in separate calls, or in one call with a strict schema and one field per question. The item's result is a vector, and the aggregate becomes a per-criterion profile that localises a regression.

Trade-off. It multiplies calls or tokens per item, and the criteria have to be genuinely independent — asking the same thing twice in different words silently double-weights it. Some properties also resist decomposition: whether a piece of writing is any good is not the conjunction of a checklist, and pretending otherwise produces outputs that pass every box and read badly.

Record the instrument with the measurement

Problem. Scores are stored as bare numbers. Six months later a comparison is made against them, after the rubric has been edited twice, the judge model has been updated by its provider, and the parsing code has changed how refusals are handled. The comparison is meaningless and nothing says so.

Approach. Write the judge model identifier and build, the rubric revision, the scale definition, the ordering policy and the scorer version onto every score as it is produced. Make the comparison tooling refuse to diff two score sets whose instrument fields differ, requiring an explicit acknowledgement instead. When the instrument changes deliberately, re-score a stored sample with both the old and the new and publish the offset between them.

Trade-off. It makes instrument changes expensive, which is the point and is also a constant source of friction — teams want to improve the rubric and now have to re-baseline to do it. The dual-scoring step is extra cost at exactly the moment somebody is trying to ship an improvement to the evaluation itself.