Evaluation and Quality Engineering
LLM-as-a-Judge — Stack
Llm as a judge › Stack
Technology landscape as of 2026-10-10. Knowledge cutoff May 2026.
Selection criteria
Two things are being chosen: the model that forms the verdict, and the scaffolding that turns a rubric into a repeatable, parseable, auditable score.
- Verdict stability. Whether the same input scored twice returns the same verdict, which decides whether the judge can be used for comparison at all.
- Structured verdict support. Whether the mechanism can constrain the output to a schema, so that parsing a score is not itself a source of error.
- Position-bias handling. For pairwise setups, whether the scaffolding swaps the order and aggregates, or leaves that to the caller to remember.
- Cost per judgement at suite scale. A reasoning-heavy judge on a two-thousand-item suite run per change is a budget line, not a rounding error.
- Build pinning. Whether the judge model can be pinned to a specific immutable build, since an unannounced provider update invalidates every stored score.
- Rubric versioning. Whether the prompt and its revision are first-class and recorded on the score, rather than living in application code.
- Human-label integration. Whether the tool makes it easy to hold out a sample, collect human verdicts on it, and report judge–human agreement as a standing metric.
- Refusal and error handling. How the mechanism distinguishes a low score from a judge that declined or failed, because conflating those two corrupts aggregates silently.
Options
| Option | Best for | Watch out for |
|---|---|---|
| A hand-written judge prompt called directly through the model API | Full control of the rubric, the schema and the aggregation, with no framework semantics in the way | Everything — ordering, retries, parsing, agreement measurement — is yours to build and to remember |
| Judge scorers bundled in evaluation frameworks (DeepEval, promptfoo, Ragas, OpenAI Evals and comparable) | Starting with rubrics that have been used before, and reporting that already exists | Default rubrics become thresholds nobody chose; the prompt is often several layers down and hard to inspect |
| Hosted judge and scoring endpoints from model and platform vendors | Offloading maintenance and getting a single number quickly | Opaque and movable rubric, no build pinning, and verdicts that shift under you without a release |
| Fine-tuned open-weight judge models (Prometheus-family evaluators and comparable published judges) | Reproducible verdicts on self-hosted infrastructure, and rubrics close to the ones they were trained on | Trained for particular rubric formats and degrade away from them; need serving infrastructure of their own |
| Reward models and preference scorers repurposed as judges | Cheap, fast relative ranking over many candidates | They encode one training population's preferences, which may not be your users', and they produce a scalar with no explanation |
| Pairwise comparison harnesses with rating aggregation (Elo-style and Bradley–Terry-style pooling) | Ranking many candidate configurations against each other reliably | Produces relative standing only; a rating cannot be compared against a release threshold or against last quarter |
| Ensembles of several judges with disagreement routed to people | High-stakes verdicts where a single judge's bias is unacceptable | Multiplies cost by the panel size, and correlated models agree with each other rather than with the truth |
Models
The property that matters most in a judge is independence from the system being judged, and it is worth more than raw capability. A judge drawn from the same family as the generator has absorbed the same preferences about structure, hedging and phrasing, and will reward its own habits; a mid-sized model from a different lineage is usually a better instrument than a larger one from the same. Second comes rubric-following rather than general ability: the judge's job is to apply somebody else's criteria faithfully, and a model that reasons its way to a more sophisticated opinion than the rubric asked for is a worse judge, not a better one. Reasoning models improve verdicts on rubrics with several interacting conditions and on anything requiring arithmetic or entailment over supplied evidence, at a cost per item that forces the suite to be smaller or the judging to be sampled — and on a simple two-way label they buy very little. Smaller and cheaper models are entirely adequate for narrow, well-anchored questions, especially equivalence checking against a reference answer, and the money saved is better spent on a larger validated sample than on a stronger judge for an easy question. Whatever is chosen, pin it to an immutable build and treat a change of build as a change of instrument: re-measure agreement against human labels before and after, and expect every stored score from the previous build to be incomparable. Note also that judge agreement with humans is not uniform across the rubric — it is usually high on obvious failures, acceptable on obvious successes, and worst in the middle band where decisions are actually made.