Evaluation and Quality Engineering
Task Completion Evaluation — Stack
Task completion evaluation › Stack
Technology landscape as of 2026-10-10. Knowledge cutoff May 2026.
Selection criteria
What matters is the environment and the inspection: somewhere the task can be performed with real consequences, and a way to read the resulting state precisely and cheaply.
- State reset fidelity and speed. Whether the environment returns to an identical starting state between items, and how long that takes, since it bounds suite size directly.
- Inspection surface. Whether the end state can be read through a stable interface — database queries, filesystem snapshots, service APIs — rather than scraped from an interface built for people.
- Negative-assertion support. Whether the inspection can assert that things are unchanged, not only that things became true.
- Isolation. Whether parallel items can run without touching each other's state, which decides whether the suite can be parallelised at all.
- Asynchrony handling. Whether the inspection can wait for eventual effects with a bounded timeout, instead of reading immediately and recording a false negative.
- Third-party realism. Whether external systems are simulated, sandboxed by the provider, or real, and what each choice costs in fidelity and in risk.
- Failure-cause separation. Whether the harness distinguishes a wrong result from a timeout, a crash, or a dependency outage.
Options
| Option | Best for | Watch out for |
|---|---|---|
| Containerised environments with snapshot and restore (container images, ephemeral databases, filesystem overlays) | Deterministic reset and parallel isolation for anything self-contained | Image drift over time; a snapshot built a year ago encodes a world the product has moved past |
| Provider sandbox environments for third-party services (payment, CRM and messaging test modes) | Realistic external behaviour without real consequences | Sandbox semantics differ from production in undocumented ways; rate limits and data retention are the provider's choice |
| Browser and interface automation (Playwright, Selenium and comparable drivers) | Tasks whose only available surface is a user interface, and reading state the way a user would | Brittle to interface change, slow, and a selector failure looks exactly like a task failure |
| Agent task benchmarks with environment and checker bundled (WebArena, τ-bench, SWE-bench-style harnesses and comparable) | A ready-made world with graded checkers, useful as a methodology reference | Their tasks are not your tasks; scores do not transfer to a product with a different state model |
| Programmatic assertion suites over the application's own data stores | Exact, fast inspection of the end state, including negative assertions | Couples the evaluation to internal schemas, which then cannot change without breaking the suite |
| Judge-based completion scoring over a trace and a final state summary | Open-ended tasks where the end state is prose or a document rather than a record | Reintroduces variance into the one measurement that could have been exact; validate against programmatic checks where both exist |
| Workflow and event-stream inspection (reading the queue, the audit log, the outbox) | Tasks whose effects are asynchronous and complete after the run finishes | Requires correlation identifiers end to end; without them, an effect cannot be attributed to the run that caused it |