Deterministic Evaluation — References

Material on checking by execution, on what surface-overlap measures actually measure, and on what happens to a measure once it becomes a target.

Checking by running it

• SWE-bench: Can Language Models Resolve Real-World GitHub Issues? (Jimenez et al., 2023), arXiv:2310.06770 — a benchmark whose success criterion is a test suite passing rather than a judgement, and the clearest demonstration of what becomes measurable once the artefact can be executed in a reset environment.

The limits of surface comparison

• The long-running critique of n-gram overlap measures for generated text — that they correlate weakly with human judgement on open-ended output, while remaining useful for tasks with a genuinely canonical reference — is well established across the machine-translation and summarisation evaluation literature. Described rather than attributed to a single title, because the result is distributed across many papers and the individual metric papers are frequently miscited.

Specifying the comparison

• Unicode Standard Annex #15, Unicode Normalization Forms — the specification of the canonical and compatibility forms, and the reason a comparison made on unnormalised text compares one encoding rather than the content. Cited by name.
• JSON Schema, as published by the JSON Schema specification project — the vocabulary most structural output checks are ultimately expressed in, and worth reading for what its keywords do and do not constrain.

Why cheap checks take over

• Goodhart's law, in the form usually quoted as "when a measure becomes a target it ceases to be a good measure", is the compact statement of the drift this capability is most prone to. Named as a principle; the original formulation was an aside in economic policy writing rather than a dedicated paper.

No verified reference is recorded for normalisation policy in evaluation comparison, for severity classification of automated checks, or for fixture-environment drift as a source of false failures.