Offline Evaluation — Stack

Technology landscape as of 2026-10-10. Knowledge cutoff May 2026.

Selection criteria

What is being bought is a runner: something that takes a set of inputs, drives the application over them, collects outputs and scores, and makes two runs comparable. Metric libraries are the easy part; the runner is where the cost is.

• Where the system under test lives. Whether the harness can invoke your actual application entry point, or only a model endpoint, decides whether retrieval and tool use are in scope at all.
• Dependency recording. Whether the tool can capture and replay external calls, and whether those recordings carry an expiry, because a replay fixture with no staleness signal is a trap.
• Concurrency and rate-limit handling. A thousand-item suite is a throughput problem, and a runner without backoff turns provider throttling into scattered failures that look like quality loss.
• Run artefact completeness. Whether a finished run stores per-item inputs, raw outputs, scores, errors and the full configuration, or only an aggregate.
• Incremental and partial runs. Whether a failed run resumes, whether a subset can be re-executed against a stored baseline, and whether cached results are reused when nothing relevant changed.
• Scoring extensibility. How hard it is to add a scorer specific to your domain, versus being restricted to the metrics that shipped.
• Determinism controls. Whether the harness pins and records decoding parameters and build identifiers itself, rather than trusting the caller to remember.
• Local-first operation. Whether a run can be executed and inspected on a developer machine without uploading anything, which decides how early in authoring the suite gets used.

Options