Deterministic Evaluation — Patterns

Make the normalisation an explicit, reviewed list

Problem. Comparison code accumulates small forgivenesses — trim the whitespace, lowercase it, drop the trailing period, ignore key order, round to two places. Each was added to stop a false failure, none was reviewed as a policy, and together they decide what the suite considers correct. Nobody can state what that is any more.

Approach. Lift every normalisation into a named, ordered list held as data, applied identically to the expected and actual values, and recorded in the run output. Adding one is a reviewed change with a stated reason. Where a normalisation is only valid for some items — unit conversion for numeric answers, say — attach it to those items rather than applying it globally.

Trade-off. Explicit normalisation is more verbose than a comparison helper with sensible defaults, and it moves friction onto the person trying to fix a flaky check at five o'clock. The list also tends to grow in only one direction, since loosening a rule fixes a visible failure and tightening one creates work.

Execute the artefact rather than inspecting it

Problem. A generated query, patch or configuration can satisfy every structural check and be wrong in a way that is only visible when it runs. Conversely, it can differ textually from the reference in a dozen ways and be exactly equivalent, which string comparison scores as a failure.

Approach. Build a fixture environment that the artefact can be executed against, and define correctness as the observable result — the rows returned, the tests passing, the resulting state matching an expected state. Reset the environment between items so results cannot depend on ordering, and cap time and resources so a pathological artefact cannot stall the suite.

Trade-off. Execution-based checking is an order of magnitude more infrastructure than text comparison, needs isolation because the artefact is untrusted, and ties the verdict to a fixture whose own drift becomes a source of false failures. The reset cost also sets a hard floor on how large the suite can be.

Keep severity attached to the check, not to the aggregate

Problem. A suite reports eighty-seven percent of checks passing. Inside that figure, a malformed response the consumer cannot parse weighs the same as an optional field being absent, so the number cannot support any decision and movements in it mean nothing in particular.

Approach. Classify every check when it is written: a violation that breaks a consumer, a violation of a stated product requirement, or a preference. Report the three separately and never average them. Allow only the first class to block automatically; the second needs a judgement, and the third exists to show a trend.

Trade-off. Classification is a standing argument, and checks migrate upward in severity over time because raising one is how a team makes a problem get attention. Three numbers are also harder to communicate than one, and the single number gets reconstructed in somebody's slide regardless.

Bound what the computable checks are allowed to represent

Problem. Deterministic checks are cheap, so they multiply, and the quality conversation drifts toward what they cover. The suite goes green while answers get worse, because nothing green was ever about whether the answer was any good.

Approach. Maintain an explicit statement of the properties the deterministic layer covers and the properties it deliberately does not, published with the results. Require that every judgement-based criterion in the quality definition has an owner and a measurement elsewhere, and treat a proposal to approximate one of them with a pattern match as a change to the quality definition rather than a tooling improvement.

Trade-off. It is a documentation discipline with no enforcement, and the pressure to replace an expensive judgement with a cheap proxy is strongest exactly when budgets are tight. Stating coverage honestly also makes the suite look weaker than a competitor's single confident number.