Agent Trajectory Evaluation — Patterns

Define checkpoints and score the segments between them

Problem. Outcome scoring gives one bit for a run of a dozen decisions, and step scoring asks a question — was this step good — that has no well-defined answer in isolation. Neither produces an attribution anybody can act on.

Approach. For each task family, define a small number of observable intermediate states that any correct run must pass through: the record was located, the eligibility was determined, the draft was produced. Score whether each checkpoint was reached, in order, and treat the steps between consecutive checkpoints as a segment whose cost and necessity can be judged. Failure then localises to the segment that did not reach its checkpoint, and efficiency becomes steps per segment rather than steps per run.

Trade-off. Checkpoints encode an assumed decomposition, and a genuinely better strategy that skips one is scored as a failure — which is the same rigidity that makes reference trajectories unusable, reintroduced at coarser granularity. They also have to be authored per task family, which is substantial domain work that nobody owns by default.

Score the step against its observation, not against a reference

Problem. The useful question about a step is whether it was a reasonable thing to do given what the system knew at that moment. Comparing to a recorded expert path answers a different question and punishes every valid alternative.

Approach. Present a judge with the state entering the step — the goal, the history so far, the observation just returned — and the action taken, and ask only whether that action was defensible from that state. The judge never sees a reference path and never sees the outcome, so it cannot score backwards from whether the run succeeded. Aggregating these per-step verdicts gives a defensibility rate, and the steps judged indefensible are the candidate causes of a failure.

Trade-off. Hiding the outcome from the judge removes hindsight bias and also removes information: an action that is locally defensible and systematically unproductive looks fine every time. Cost scales with trajectory length rather than with run count, which makes this the most expensive scoring mode in the stack.

Separate error from non-recovery

Problem. Treating any wrong step as a defect pushes systems toward not visibly erring, which in practice means not checking, not retrying, and not noticing contradictions. The runs that recover from a bad observation are the ones that are working, and naive step scoring marks them down.

Approach. Classify each run on two axes: whether a wrong or unproductive step occurred, and whether the run subsequently detected and corrected it. Report the four combinations. The population that matters is runs where an error occurred and was not corrected; runs that erred and recovered are tracked for their cost, not scored as failures. Changes are then judged on whether they move runs from the uncorrected cell to the corrected one.

Trade-off. Detecting that a recovery happened requires a judgement about intent, which is exactly the thing traces represent poorly, and a system that loops repeating a failing call will be scored as recovering. The two-axis report is also easy to collapse back into one number by anyone preparing a summary.

Replay from a divergence point rather than from the start

Problem. Re-running a failed task from the beginning produces a different trajectory, often one that succeeds, which tells you nothing about the original failure. Full replay of the recorded run reproduces it exactly and tests nothing, because every decision is canned.

Approach. Replay the recorded observations deterministically up to the step under investigation, then let the system decide freely from that state with the fix applied. The comparison is between the recorded continuation and the new one from an identical starting state, which isolates the change. Keep the divergence point as part of the test case, so the scenario becomes a permanent regression item rather than a one-off investigation.

Trade-off. It requires the environment to accept a mid-run state injection, which most real external systems do not, and a replayed prefix can put the system in a state the current code would never have produced. The resulting test cases also rot quickly, since any change to the prompt or the tool surface can make the recorded prefix unreachable.