Offline Evaluation — Patterns

Freeze the dependencies, not just the inputs

Problem. A suite whose inputs are fixed but whose retrieval index, tool backends and model build are live is measuring the whole world. A score that falls on Tuesday might be a prompt change, a reindex, a rate-limited third party, or a silently updated endpoint, and nothing in the result distinguishes them.

Approach. Capture the external interactions once and replay them, so that the only live component is the part under test. Store the recordings beside the suite, stamp each with when it was captured and against what, and fail the run loudly when a request arrives that has no recording rather than falling through to the network. Keep a second, slower configuration that runs everything live, and schedule it on its own cadence to detect drift between the recording and reality.

Trade-off. Replay makes the suite fast, cheap and attributable, and makes it progressively less true. The recordings are a second dataset with its own staleness problem, and the moment anyone trusts a replayed pass as evidence that the live system works, the pattern has done harm.

Report per-item verdicts as the primary artefact

Problem. An aggregate score compresses a thousand independent observations into one number, and the number is the least informative function of them. Two candidates can tie at 0.72 while disagreeing on a third of the set, and the team picks one by coin flip.

Approach. Make the run emit a stable, sorted record per input — identifier, output, score, scorer rationale, errors — and treat the aggregate as a derived view over that. Diffing two runs then becomes a set operation that names which inputs changed verdict in which direction. Identifiers must be stable across dataset edits for this to work, which forces item identity to be designed rather than positional.

Trade-off. Per-item artefacts grow fast, need storage and retention policy, and often contain production-derived content with the privacy obligations that implies. There is also a discipline cost: the artefact is only useful if somebody reads the diff, and the aggregate is always easier to paste into a message.

Hold out a slice nobody is allowed to read

Problem. Any set that engineers inspect while authoring becomes a target. Failing inputs get turned into few-shot examples, instructions get special cases, and the suite climbs toward a perfect score while the behaviour on unseen requests does not move at all.

Approach. Split the set at curation time into a development portion that authors may read freely and a reserved portion whose individual items are never shown to anyone writing the system. Only aggregate numbers come back from the reserved portion. Rotate items between the two on a schedule, since a reserved slice that is scored a hundred times has been fitted through the aggregate alone.

Trade-off. Reserving items halves the data available for debugging, which is where most of a suite's day-to-day value comes from, and the discipline is unenforceable by tooling — one engineer reading reserved failures during an incident quietly destroys the property for everyone.

Make a run a declared configuration, not a command line

Problem. Runs are started from shell history with flags that differ between people, so two scores that look comparable were produced with different concurrency, different decoding settings, and in one case a truncated subset of the data.

Approach. Express the run as a single checked-in configuration object naming the dataset identity, the system entry point, the dependency mode, the decoding parameters, the scorers and their versions. The runner takes that object and nothing else, and copies it verbatim into the result. Comparing two runs starts by diffing their configurations, and the harness refuses to diff scores when the configurations differ in a field that affects results.

Trade-off. It adds friction exactly where people want speed, and the pressure to add an override flag "just for this one run" is constant. The configuration also becomes a schema that has to be migrated, and old results become unreadable when it changes carelessly.