LLM-as-a-Judge

What it is

Using a language model to score or rank another model's output against a written rubric. The judge reads the input, the response, and whatever reference material the rubric requires, and emits a verdict — a label, a score on a scale, or a preference between two candidates. It is the only scoring mechanism that can read meaning, which is why it is used at all, and it is a model with its own errors, which is why its verdicts are a measurement with a known bias rather than ground truth.

Why it is not a cheap substitute for a person

The comparison that matters is not cost per item but what each mechanism is sensitive to. A judge applies the same rubric to the ten-thousandth item exactly as it did to the first, which no person does, and it will apply a subtly wrong reading of that rubric ten thousand times with equal consistency, which no panel of people would. People disagree with each other and notice when a question is badly posed; a judge never objects to the rubric, it just returns a number. So a judge is best understood as an amplifier: it takes a small amount of human judgement, encoded in a rubric and validated against human labels, and applies it at a scale humans cannot reach. Without that validation step it amplifies nothing and manufactures confidence instead.

Why it matters

Most of what an application is judged on is not computable. Whether an answer addressed the question, whether the tone suits a complaint, whether a summary kept the point and dropped the noise, whether an explanation would help a novice — none of these reduce to a string comparison, and all of them are what users actually react to. Before model judging, the options were a proxy metric that correlated weakly with the property of interest, or a human review cycle measured in days. A validated judge makes open-ended quality something that can be measured on every candidate change rather than once a quarter.

Key decisions

• Whether the verdict is absolute or comparative. Scoring one response on a scale invites score clustering and drifts between runs; comparing two responses is far more stable and gives you a ranking rather than a level, which cannot be compared to last quarter's number.
• What the rubric lets the judge see. Withholding the reference answer tests whether the response stands on its own; supplying it collapses the task to equivalence checking and makes the judge far more reliable and far narrower.
• How the judge is validated. Agreement with human labels on a reserved sample is the only evidence that the verdicts mean anything, and that agreement has to be re-measured whenever the rubric, the judge model or the output distribution changes.
• Whether the judge is required to produce its reasoning, which improves verdicts on multi-part rubrics and makes every evaluation several times more expensive.
• Whether the judge may come from the same family as the system it scores, given that it will recognise and reward its own characteristic phrasing.
• How the scale is defined. A one-to-five scale with undefined midpoints produces threes; a scale with a written anchor for every point produces usable distinctions and takes real effort to author.
• Whether a judge model identifier and rubric revision are stored with every score, since without them a historical comparison spans two different instruments.

Failure modes

• A rubric is rewritten for clarity, every score shifts by a few points, and the shift is attributed to the release that happened the same week.
• The judge prefers whichever candidate it was shown first, and the comparison measures ordering rather than quality, because nobody randomised the positions.
• Longer, more confident, better formatted answers score higher regardless of whether they are correct, and the system is gradually tuned toward verbosity.
• The judge is from the same family as the generator, systematically rates its own style as better, and the evaluation quietly blocks every alternative model.
• Judge agreement with humans was measured once at setup, at seventy percent, and is treated thereafter as if it were a hundred.
• The hosted judge model is updated by its provider, every score moves, and nothing in the evaluation records which build produced which number.
• The judge is asked to score five dimensions in one call and returns five nearly identical numbers, because it formed one overall impression and distributed it.
• Scores are averaged across a set where the judge returns a hard refusal on a handful of sensitive items, and those refusals are silently counted as zeros.
• The judge sees only the response and not the tool output it was summarising, so it rates a fluent misstatement of the data as excellent.

Related capabilities

• Human Evaluation — the source of the labels a judge has to be validated against
• Deterministic Evaluation — the cheaper mechanism to prefer wherever the property is computable
• RAG Evaluation — the setting where a judge is given explicit evidence and asked about support
• Confidence Calibration — what to ask of a judge that reports its own certainty
• Evaluation Gates — the decision that inherits every bias in these verdicts

Applies: Production readiness · Design questions