Model Management and Inference

Model Management and Inference

Model Benchmarking

46 of 49 in this book · 3 min read

What it is

Measuring candidate models against each other on tasks drawn from your own workload, to decide which to adopt, which to route where, and whether an upgrade is safe. The subject under test is the model: prompt, configuration, harness, and dataset are held fixed while the model varies. That is what separates it from evaluating a feature, where the model is fixed and the prompt or pipeline is what moves.

Why it matters

Published leaderboards answer a question nobody in a product team has. They report average performance on academic tasks, under prompts tuned by the submitting party, at a concurrency of one, with no mention of what a completed task costs. Model choices made from them are routinely reversed within weeks of real traffic. A benchmark built from your own tasks, scored on the properties your product actually depends on, is the only thing that makes "model A is better for us" a finding instead of an opinion.

What it measures that a leaderboard does not

Quality is one axis among several, and rarely the binding one. Latency at the concurrency you serve, cost per successfully completed task rather than per token, the rate at which structured output parses on the first attempt, the rate of unnecessary refusals on benign inputs, and behaviour at the long end of your input length distribution are all model properties, all vary widely between models that score similarly, and all show up in production immediately.

Key decisions

  • What the benchmark is for. Selecting a new model, gating a version upgrade, and sizing hardware are three different measurements, and a suite built for one is usually misleading for the others.
  • How cases are sampled from production. Uniform sampling measures the easy middle; deliberate stratification by task type, input length, and known-hard categories measures what breaks.
  • How correctness is judged: exact match, schema validation, programmatic checks against a ground truth, human review, or a model as judge — and for the last, how judge bias toward its own family is controlled.
  • Whether the prompt is held identical across candidates or tuned per candidate. Identical is reproducible and unfair to models with different conventions; tuned is fair and no longer a controlled comparison.
  • Sample size, and what difference the suite can actually detect. Reporting a two-point gap from forty cases is reporting noise.
  • How contamination is handled — whether cases could plausibly appear in training data, and whether any private cases are kept unpublished as a control.

Failure modes

  • Quality is measured at concurrency one, the model is deployed under real load, and the latency that informed the decision bears no relationship to the latency users see.
  • The prompt was developed against the incumbent model over months, and the challenger is evaluated with it; the comparison measures prompt fit, not capability.
  • Cost is reported per token. A model with a lower token price that needs two attempts to produce valid output is more expensive per completed task and the benchmark says otherwise.
  • The suite becomes a fixed target. Prompts and pipelines are tuned against it until scores rise and production quality does not.
  • A model-as-judge scores a candidate from its own family higher, and the bias is never checked against human labels on a sample.
  • Cases were drawn from public sources that the candidates trained on, so the benchmark measures memorisation.
  • Results are stored without the model version, prompt version, and configuration hash, making the historical record unusable as a baseline six months later.
  • Only aggregate scores are reviewed, so a model that is better on average and much worse on one critical task type is adopted.

Related capabilities

Applies: Production readiness · Design questions