Deterministic Evaluation

What it is

Scoring by computation rather than by judgement: a check that reads the output, applies a fixed rule, and returns the same verdict every time. The family covers exact and normalised matching, schema and type validation, pattern presence and absence, numeric comparison within a tolerance, constraint satisfaction, and executing something the output produced — running the generated query, compiling the generated code, applying the generated patch and running the test suite. What unites them is that nothing in the scoring path has an opinion, so two runs disagree only if the system did.

Why it matters

A check with no variance is the only kind that can carry a release decision without a confidence interval attached. It costs microseconds rather than a model call, which means it can run on every item of every suite and on every request in production if wanted. And a surprising proportion of what actually breaks in production is structural rather than semantic: malformed JSON that fails the parser, a required field absent, a citation marker pointing at a passage that was not retrieved, a currency where a date belonged. These defects are cheap to catch and expensive to discover downstream, and no amount of judgement-based scoring finds them more reliably than a parser does.

Key decisions

• How much normalisation happens before comparison — whitespace, case, punctuation, key ordering, numeric formatting, unit conversion — since every normalisation rule widens what counts as correct and each one is a small policy decision.
• Whether the check tests the artefact or executes it. A generated query that parses may still be wrong; running it against a fixture database is a far stronger check and needs an environment to run in.
• What tolerance applies to numeric comparison, and whether it is absolute or relative, because the wrong choice makes small values always pass or always fail.
• Which properties are genuinely computable and which only look it. Whether a summary covered the key points is not a pattern match, and encoding it as one produces a precise measurement of something else.
• Whether a failed check reports which rule failed and on what fragment, or only that the item did not pass, since the second is nearly useless during an investigation.
• Whether checks are expressed as data that a non-engineer can author and review, or as code, which determines who can add one when a new defect class appears.

Failure modes

• A correct answer fails because it wrote the number with a thousands separator, and the team loosens the check until it no longer distinguishes anything.
• Normalisation strips enough that two genuinely different outputs compare equal, and a real regression passes.
• Pattern matching stands in for a semantic property — a regular expression for "mentions the refund policy" — and the system learns to emit the phrase without the substance.
• The check validates a schema that the application has since changed, so it passes outputs the consumer rejects.
• Relative tolerance is applied to values near zero and everything passes; absolute tolerance is applied to values in the millions and everything fails.
• Execution-based checking runs against a fixture environment whose state has drifted, and queries that are correct now fail against data from last year.
• The suite is reported as a single pass rate across checks of wildly different severity, so a broken schema and a missing optional field contribute equally.
• Cheap checks crowd out the hard questions: everything measurable is measured, the measurable portion becomes the definition of quality, and nobody notices that the answers got worse while every structural check stayed green.
• A check is written against one model's characteristic output format and silently fails every candidate replacement for reasons unrelated to quality.

Related capabilities

• LLM-as-a-Judge — the mechanism to reach for only where the property resists computation
• Decision Model Evaluation — where exact comparison against a label is the whole method
• Task Completion Evaluation — depends on these checks to inspect an end state
• Evaluation Gates — the natural home for a check with no variance
• Regression Evaluation — benefits most from scorers that cannot themselves move

Applies: Production readiness · Design questions