Evaluation and Quality Engineering
Deterministic Evaluation — Stack
Deterministic evaluation › Stack
Technology landscape as of 2026-10-10. Knowledge cutoff May 2026.
Selection criteria
Every option here is a way of expressing a rule and running it. The choice is governed by who writes the rules, how precisely a failure is localised, and what has to exist for the rule to be checkable at all.
- Failure localisation. Whether a failure names the specific rule and the offending fragment, or returns a single boolean for the whole item.
- Authoring audience. Whether rules live in code, in a declarative file, or in a form a domain expert can edit.
- Normalisation control. Whether the comparison's normalisation is explicit and configurable, rather than hidden inside a library's defaults.
- Execution sandboxing. For checks that run generated code or queries, whether execution is isolated, resource-limited and reset between items.
- Environment reset cost. How quickly a fixture database, filesystem or service can be returned to a known state between checks, since this dominates the runtime of execution-based suites.
- Determinism under concurrency. Whether running the suite in parallel can change any verdict, which it will if checks share mutable fixture state.
- Schema evolution. Whether validation rules can be generated from the same definitions the application uses, so they cannot drift apart.
Options
| Option | Best for | Watch out for |
|---|---|---|
| Schema validators driven by the application's own definitions (JSON Schema, Pydantic, protobuf and equivalents) | Structural correctness of anything the system is supposed to emit in a defined shape | Validates shape, never meaning; a perfectly typed object can be entirely wrong |
| Assertion libraries inside the existing test framework | Checks that engineers write and maintain beside the code they test | Framework assertions stop at the first failure, which hides the other four problems in the same item |
| Declarative check definitions in evaluation frameworks (promptfoo-style assertion blocks, DeepEval's non-model metrics and comparable) | Letting non-engineers add and review checks without touching application code | Expressiveness ceiling is reached quickly, and the escape hatch is custom code anyway |
| Grammar and constrained-decoding validators (parser generators, structured-output constraint libraries) | Checking that output conforms to a formal language, and sometimes preventing the violation outright | Conformance to a grammar is a weak guarantee; the parse succeeding says nothing about the content |
| Execution-based checking against a sandbox (containerised runners, ephemeral databases, test-suite harnesses) | Generated code, queries and transformations, where running the artefact is the strongest check available | Needs isolation, resource limits and a reliable reset; environment drift silently inverts results |
| Text-overlap and similarity measures (exact match, normalised edit distance, token overlap statistics, embedding similarity) | Cheap screening and regression detection where the reference answer is genuinely canonical | Reproducible but not meaningful on open-ended output; embedding similarity is reproducible and is not deterministic in the sense that matters here |
| Property-based and invariant checking (generators that assert a relationship holds across many inputs) | Rules of the form "whatever it answers, this must remain true" — units preserved, totals adding up, no entity invented | Writing good invariants is hard, and a weak invariant passes everything while looking rigorous |