Offline Evaluation

What it is

Running the application over a held-out set of inputs, before the change under test reaches anybody, and scoring what comes back. The inputs do not move between runs, and that immobility is where the whole method gets its power: two numbers are comparable because everything except the change was held still. Most of the engineering is harness work rather than metric work — deciding how much of the real system executes, what is replaced by a recording, what is pinned, and how long the run is allowed to take.

Why a frozen input set is the point

Holding the inputs fixed buys attribution. When a score moves and nothing else was allowed to move, the change under test is the explanation, and that is a claim no measurement on live traffic can make without randomisation. The price is paid immediately and forever: a set frozen for comparability stops resembling the requests people actually send, and the gap widens every week nobody touches it. Every design choice here is some version of that trade, and a team that never refreshes its set has chosen one side of it by default.

Why it matters

A prompt edit, a model build, a retrieval parameter or a reworded tool description can degrade behaviour nobody thought to check, and the alternative to catching that before release is hearing about it from the people it happened to. It is also the only instrument available to a system with no users yet, which is exactly when the most consequential design choices are made. And because the run is repeatable on demand, it is the only place where a proposed change can be compared against three alternatives in an afternoon instead of three weeks of staged rollout.

Key decisions

• How much of the real system executes. A model call with a recorded context isolates the generation step and runs in seconds; the full request path exercises retrieval, tools and assembly, and inherits every one of their outages into the result.
• Whether the run is forced to be reproducible — pinned build, fixed decoding parameters, seeded sampling — or is run several times and reported as a distribution. The first hides variance the product actually has; the second multiplies the cost of every evaluation.
• What the suite's output is. An aggregate number ranks two candidates and explains nothing; a per-item verdict file explains everything and nobody reads it; most teams need both and build only the first.
• The runtime and cost budget, because that single figure decides whether the suite runs on every change, on every release candidate, or whenever someone remembers.
• How difficulty is distributed across the set. A suite on which everything passes has no resolution left to detect improvement, and a suite on which everything fails cannot detect regression either.
• Who is allowed to look at individual failing inputs while authoring a change, since a set that is read during development is being fitted rather than measured against.

Failure modes

• Dependencies are replaced by recordings captured when the harness was built, so the suite faithfully measures the behaviour of a system that stopped existing nine months ago.
• Decoding is left at the product's sampling settings, the same build scores two points apart on consecutive runs, and a real two-point regression is indistinguishable from that noise.
• The suite takes forty minutes and costs enough to notice, so it runs once at the end, when the only remaining options are ship it or slip the date.
• Expected outputs were written by the person who wrote the prompt, in the same sitting, so the set encodes that person's reading of the requirement rather than the requirement.
• Every input that fails gets pasted into the prompt as an example until the suite passes completely and generalises to nothing.
• The run executes against an endpoint whose underlying build was updated between the baseline and the candidate, and the comparison silently spans two different systems.
• Scores are stored without the configuration that produced them, so last quarter's number cannot be reproduced and cannot be argued with.
• The harness swallows errors from a dependency and records them as low scores, so an outage in a retrieval service is reported to the team as a quality regression in the model.

Related capabilities

• Golden Datasets — the curated inputs and expected behaviour this harness consumes
• Online Evaluation — the measurement that begins where a fixed set stops being representative
• Regression Evaluation — what to do with two runs of this suite that disagree
• Evaluation Dataset Versioning — how a stored score stays attached to the inputs that produced it
• Deterministic Evaluation — the cheapest scoring mechanism a run of this kind can use

Applies: Production readiness · Design questions