Evaluation and Quality Engineering
Agent Trajectory Evaluation — Stack
Agent trajectory evaluation › Stack
Technology landscape as of 2026-10-10. Knowledge cutoff May 2026.
Selection criteria
Two things have to exist before any scoring is possible: a faithful record of the run, and an environment the run can be reproduced against. Everything else is downstream of those.
- Trace completeness. Whether the record captures every decision, every call with its arguments, every observation returned, and the state at each boundary — a trace missing the observations cannot support attribution.
- Step identity across runs. Whether two runs of the same task can be aligned step by step for comparison, given that they will diverge in length.
- Environment reproducibility. Whether the external world the run acted on can be reset to a known state and replayed, which decides whether a failure can be investigated at all.
- Nested and parallel structure. Whether the representation handles sub-agents, parallel branches and retries, rather than assuming a flat sequence.
- Segment annotation. Whether a reviewer or scorer can attach a verdict to a span of steps rather than only to the whole run or to individual steps.
- Cost and latency attribution per step. Whether the trace carries tokens, money and time per step, since efficiency claims are unfalsifiable otherwise.
- Trace volume economics. Agentic runs produce one to two orders of magnitude more trace data per task than single-turn products, which changes retention arithmetic.
Options
| Option | Best for | Watch out for |
|---|---|---|
| Agent-aware tracing in LLM observability platforms (LangSmith, Langfuse, Arize Phoenix, Weights & Biases Weave and comparable) | Seeing a run as a tree of steps with inputs, outputs and costs, and annotating spans by hand | Scoring is usually per-run or per-span; cross-run alignment and segment aggregation are yours |
| OpenTelemetry spans with generative-AI conventions into a general backend | Keeping agent runs beside the rest of the system's telemetry and reusing existing retention policy | Attribute size limits truncate large observations; sampling defaults drop exactly the long runs you need |
| Agent benchmark environments with built-in task and trajectory scoring (WebArena, AgentBench, τ-bench and comparable research harnesses) | A reproducible world to run against, with tasks whose success is checkable | Their task distribution is not yours; useful as a methodology reference more than as a measurement of your product |
| Record-and-replay of the environment (HTTP cassettes, containerised service fixtures, seeded databases) | Making a failed run reproducible and re-scorable after the fact | Replay diverges the moment the agent takes a different action than it took when recording |
| Trajectory-scoring metrics in evaluation libraries (DeepEval's agent metrics, Ragas agent scorers and comparable) | Off-the-shelf notions of step correctness and goal alignment to start from | Definitions vary and are rarely documented precisely; validate what a metric counts before reporting it |
| A simulated environment built for your own domain | Products where the real external systems cannot be exercised safely or cheaply | The simulator becomes a system with its own fidelity problem, and agents overfit to its quirks |
| Hand-built trajectory diff and review tooling over stored traces | Investigations that need two runs of the same task side by side, step aligned | Alignment of divergent sequences is a genuinely hard problem; naive positional diffs are misleading |