Task Completion Evaluation — Patterns

Write the end state as assertions before the task is attempted

Problem. Completion judged after the fact drifts to whatever the reviewer notices. The same run is passed by one person and failed by another, and the disagreement is really about what the request meant — a question nobody answered before the run.

Approach. Author each task as a pair: the request as the user would phrase it, and a separate machine-checkable statement of the end state that makes it complete. Write the second before seeing any system output, so it describes the requirement rather than the behaviour. Review the pair as a unit, since most arguments about a failing item turn out to be arguments about whether the assertion was right.

Trade-off. Authoring end-state assertions is slow, needs someone who understands both the domain and the data model, and produces a suite that is expensive to extend. The assertions also over-specify: they pin a particular correct outcome, and a differently shaped but equally acceptable result fails.

Assert what must not change

Problem. A specification that lists only the required effects is satisfied by any run that produces them, including one that also cancelled an unrelated order, emailed the wrong customer or rewrote a configuration file. These are the failures that generate incidents, and a positive-only check cannot see them.

Approach. Capture a digest of the relevant environment state before the run and compare after, treating any difference outside the declared allowed set as a failure. Scope the comparison to the blast radius the task could plausibly touch, since a whole-environment diff is dominated by timestamps and incidental writes. Report unexpected changes separately from missing required ones, because they indicate different defects.

Trade-off. Defining the allowed change set is finicky and needs maintenance as the application writes new things, and a diff scoped too widely produces constant false failures that train everyone to ignore it. Scoped too narrowly, it misses precisely the surprising damage it was built to find.

Separate did-not-finish from finished-wrongly

Problem. A single failure bucket mixes outcomes with unrelated causes: the system ran out of steps, the environment timed out, a dependency was down, or it confidently did the wrong thing. Pooled, they produce a completion rate that moves for reasons nobody can attribute.

Approach. Classify every non-success at the moment it occurs into exhausted, errored, abandoned, or completed-incorrectly, using signals the harness already has — step limits, exceptions, explicit give-up messages, assertion results. Report the four as separate series. Infrastructure problems then show up in their own line rather than as a quality regression, and the completed-incorrectly series is the one that actually tracks model behaviour.

Trade-off. The classification is heuristic at the edges — a system that loops until the step limit after a wrong decision is arguably both exhausted and incorrect — and the categories need a tie-breaking rule that will be arbitrary. Four series are also easier to ignore than one number, and the headline completion rate gets quoted regardless.

Grade with weighted sub-goals rather than one bit

Problem. Realistic tasks have several components, and a binary verdict throws away the difference between a run that did nothing and one that did everything except the final confirmation. With a binary metric, a change that improves most of the work on most tasks is invisible.

Approach. Decompose each task's end state into named sub-goals with weights reflecting their importance, check each independently, and report both the fraction of tasks fully complete and the weighted partial score. The first is the honest headline, because a partly done task is usually worthless to the user; the second is the sensitive instrument that shows whether work is moving in the right direction between releases.

Trade-off. Weights are an invented quantity and will be argued over, and a weighted score creates an incentive to optimise the cheap sub-goals. Partial credit can also mislead badly in domains where an incomplete action is worse than no action — a half-executed transfer is not half a success.