Task Completion Evaluation

What it is

Deciding whether the thing the user asked for actually got done. The task arrives as a sentence, the result is a change in the world — a record updated, a message sent, a file written, a booking made — and the measurement is an inspection of that world afterwards against a statement of what should have been true. Almost all of the difficulty is in writing that statement: a natural-language request rarely specifies what counts as finished, and two reasonable people reading the same request will inspect for different things.

Why it matters

Every other measurement in an agentic product is a proxy for this one. A system can select the right tools, follow a defensible path, produce well-grounded prose and leave the user's actual job undone, and nothing short of inspecting the end state will say so. It is also the only measurement that catches the two failures users complain about most: the task reported as complete when it was not, and the task completed along with three side effects nobody asked for. Both are invisible to any scoring that reads only what the system said about itself.

Key decisions

• Where the definition of done lives. A per-item written specification of the required end state is the only durable form; anything less means the criterion is reconstructed by whoever reviews the run, differently each time.
• Whether completion is binary or graded. Many real tasks finish partially, and a binary verdict either rewards a half-finished job or discards the distinction between ninety percent and nothing.
• Whether side effects are in scope. A specification that lists only what must be true permits everything else; a specification that also states what must be unchanged is several times more work and is the only way unrequested damage is detected.
• How the system's own claim of completion is treated — as the thing being scored rather than as evidence, since self-reported success is exactly the quantity most likely to be wrong.
• Whether the end state is inspected programmatically or judged. Programmatic inspection is exact and requires the environment to expose its state; judgement handles open-ended outputs and reintroduces variance.
• What environment the task runs in and how it is reset, because an inspection is only meaningful against a known starting state and most tasks are not idempotent.
• Whether the goal being scored is the one stated or the one intended, given that a literal reading of an ambiguous request can be satisfied perfectly and be useless.

Failure modes

• Completion is recorded from the system's closing message, which says the ticket was updated, and nobody queried the ticket.
• The specification lists what must become true and says nothing about what must stay true, so a run that achieves the goal by deleting three unrelated records passes.
• Tasks run against a shared environment without reset, so an item passes or fails depending on what the previous item left behind.
• A request was ambiguous, the system chose a defensible reading, and the item is scored against the reviewer's different reading — which turns a specification defect into a reported model regression.
• Partial completion is scored as failure, so a change that makes the system stop early rather than attempt the hard half looks neutral in the aggregate.
• The suite contains only tasks whose end state is cheap to inspect, so coverage silently concentrates on a narrow and unrepresentative class of work.
• Verification queries the same cached layer the system wrote through, so a write that never reached the system of record reads back as present.
• Timeouts and resource exhaustion are recorded as task failures without being distinguished from wrong answers, and an infrastructure problem is debugged as a quality problem for a week.
• A task is scored as complete at the moment the system stops, while the real work completes asynchronously minutes later and sometimes does not.

Related capabilities

• Agent Trajectory Evaluation — decomposes a completion failure into the step that caused it
• Deterministic Evaluation — the inspection mechanism that makes an end-state check exact
• Golden Datasets — where per-task end-state specifications are curated and reviewed
• Multi-turn Evaluation — the case where the goal is established across several exchanges rather than stated once
• Online Evaluation — measures the same question on live traffic, where no specification exists

Applies: Production readiness · Design questions