Task Completion Evaluation — References

Sources on tasks whose success is read from the environment rather than judged, and on the benchmark designs that made that possible.

Checkable end states

• SWE-bench: Can Language Models Resolve Real-World GitHub Issues? (Jimenez et al., 2023), arXiv:2310.06770 — a benchmark whose success criterion is a test suite passing rather than a judgement, and the clearest demonstration of what becomes measurable when completion can be inspected instead of assessed.

Environment-based task benchmarks

• The interactive-environment benchmark line — tasks specified in natural language against a simulated website, operating system or service, with success read from the resulting environment state and with explicitly authored checkers per task — is the reference design for this capability. Described as a body of work rather than enumerated by title, because several of these projects share names across distinct papers and successive versions.
• The agent-against-service-API benchmarks that pair a user simulator with a policy document and check the resulting database state are the closest published analogue to a business task with side effects. Described rather than named precisely, to avoid attributing a title inaccurately.

Why self-reported completion is not evidence

• Reflexion: Language Agents with Verbal Reinforcement Learning (Shinn et al., 2023), arXiv:2303.11366 — relevant here for the gap it exploits between what a run concluded about itself and what actually happened, which is the gap an end-state inspection exists to close.

No verified reference is recorded for negative assertions over environment state, for the exhausted-errored-abandoned-incorrect classification used here, or for weighted sub-goal grading of business tasks.