Agent Trajectory Evaluation

What it is

Scoring the path a multi-step run took, not only where it ended up. A trajectory is the ordered record of what the system decided, what it called, what came back, and what it concluded at each turn, and this capability asks whether that sequence was a reasonable way to get where it got — whether the work was necessary, whether the steps followed from the observations, and whether the run's length and cost were proportionate to the task.

The credit-assignment problem

A run of eleven steps ends in a wrong answer and produces exactly one outcome bit. The bit says nothing about which of the eleven steps caused it, and in practice the causative step is rarely the last one: a misread observation at step three propagates through eight subsequent decisions that were each locally sensible. Comparing against a reference trajectory does not solve this, because most tasks have many correct paths and a run that took a different good one is scored as wrong. Scoring each step independently does not solve it either, because a step's quality is conditional on the state it inherited, and a recovery step looks bad in isolation precisely when it is doing the most valuable work in the run. Everything in this capability is an attempt to get attribution out of a signal that does not naturally carry it.

Why it matters

Outcome-only measurement cannot distinguish a system that reasons from one that is lucky, and the difference shows up the moment the task distribution shifts. It also leaves the two cost drivers of agentic products — redundant steps and unnecessary escalation to expensive calls — entirely unmeasured, so a change that doubles the average step count while holding success constant looks free. And when a run does fail, the backlog item that comes out of it is only actionable if somebody can say which decision was the wrong one; without trajectory-level measurement, every agent failure is reported as "the agent got confused".

Key decisions

• Whether the unit of score is the outcome, the step, or a segment between checkpoints. Segment-level scoring is the usual compromise and requires defining checkpoints that most tasks do not naturally have.
• Whether a reference trajectory exists at all, and if so whether it is treated as the required path or as one acceptable path among many.
• How recovery is credited. A run that erred and corrected itself is better than one that never erred only in that it demonstrated resilience, and worse than it in cost — the scoring has to take a position.
• What counts as progress, since efficiency metrics are meaningless without it: steps per task is a cost measure, not a quality one, until you can say which steps advanced the goal.
• Whether the environment the trajectory ran against is reproducible, because a trajectory recorded against live external services cannot be re-scored or replayed.
• How much of the model's intermediate reasoning is captured and whether it is scored, given that stated reasoning need not correspond to the decision actually taken.
• Whether failed runs are sampled for review at a higher rate than successful ones, and how the resulting bias is accounted for in any aggregate.

Failure modes

• Scoring is outcome-only, so a run that reached the right answer by ignoring its tools and guessing is indistinguishable from one that reasoned correctly.
• Every trajectory is compared to one recorded expert path, and a shorter, better route is penalised for deviating at step two.
• Step-level scoring marks a correction as a defect, and a change that suppresses corrections improves the metric while making failures permanent.
• Efficiency is reported as mean step count, which falls because the hardest tasks started failing early and short.
• Trajectories are recorded against live external services, so a failure cannot be reproduced and the investigation ends in speculation.
• The captured trace omits the tool results, so a reviewer can see the system made a strange decision and not that it was handed a malformed observation.
• Only failures are reviewed, and the review population has no idea how often the same odd behaviour appears in runs that happened to succeed.
• Stated reasoning is scored as though it were the decision process, and a model that produces convincing rationales for arbitrary actions scores well.
• The aggregate pools runs of wildly different task difficulty, so a change in the task mix moves every trajectory metric without anything in the system changing.

Related capabilities

• Task Completion Evaluation — the outcome question this capability exists to decompose
• Tool Selection Evaluation — scores one decision type that appears repeatedly inside a trajectory
• Multi-turn Evaluation — the analogous problem when the other participant is a person rather than an environment
• Human Evaluation — where a disputed trajectory is adjudicated step by step
• LLM-as-a-Judge — the scorer usually asked whether a step followed from its observations

Applies: Production readiness · Design questions