Model Benchmarking — Patterns

A stratified task set drawn from real traffic

Problem. Cases assembled by hand reflect what the author imagined. Cases sampled uniformly from production reflect the easy middle of the distribution, where every candidate model does well and no decision is informed.

Approach. Stratify the sample along the axes that predict difficulty for your workload — task type, input length, language, domain vocabulary, presence of ambiguity, known historical failures — and sample within each stratum, deliberately over-weighting the hard strata. Report scores per stratum as well as in aggregate, and treat a per-stratum regression as blocking even when the aggregate improved.

Trade-off. The aggregate no longer represents production frequency, so it cannot be read as an expected production score without reweighting. Maintaining strata is ongoing work, and the strata themselves encode last quarter's understanding of what is hard.

Quality and load measured as one decision

Problem. Accuracy is measured by a correctness harness running serially, and latency by a load test using a trivial prompt. Both numbers are real and neither describes the model answering your traffic under your concurrency.

Approach. Run the benchmark twice over the same case set: once serially for correctness with full output capture, and once as a replay at the target request rate and concurrency, recording time-to-first-token, total latency, error and throttle rates, and token usage per case. Present the result as a frontier — quality against cost and latency — rather than as a ranked list, because the choice is a trade and the decision maker should see it as one.

Trade-off. Two runs cost roughly twice as much, and the load run against a hosted provider measures a shared system whose behaviour depends on other tenants and on the time of day. Repeat it at different times before treating a latency gap as a model property.

Cost per completed task, not per token

Problem. Token prices are the visible number and the misleading one. A cheaper model that fails schema validation and needs a retry, or that produces three times the output to say the same thing, costs more per unit of work delivered.

Approach. Define completion as the product defines it — valid parseable output, a correct extraction, a resolvable citation — and compute total tokens spent per completed case, including every retry, repair attempt, and escalation the pipeline would have performed. Report that figure per candidate alongside the quality score.

Trade-off. It requires the benchmark to simulate the production retry and repair behaviour rather than making a single clean call, which makes the harness meaningfully more complex and couples it to pipeline changes. Without that simulation the number is a token count wearing a cost label.

Held-out private cases as a contamination control

Problem. Cases drawn from public sources may be in a model's training data, and a suite that is iterated against week after week is gradually optimised for, whether or not anyone intends it.

Approach. Split the case set into a working portion used freely during development and a sealed portion run only at decision points — model selection, upgrade gating — and never inspected case by case. Draw the sealed portion from material that was never published, and rotate a fraction of it on a schedule. A large gap between working and sealed scores is the contamination signal.

Trade-off. Sealed cases are a wasting asset: every time they are run and discussed they leak a little into the next iteration, and rotation requires continuously producing fresh unpublished cases. The control costs real effort and only ever gives you a warning sign, not a clean measurement.