Evaluation and Quality Engineering
Multi-turn Evaluation — Stack
Multi turn evaluation › Stack
Technology landscape as of 2026-10-10. Knowledge cutoff May 2026.
Selection criteria
The distinguishing requirement is a counterpart: something that produces the next user message in response to what the system actually said. Everything else follows from how that counterpart is built and controlled.
- Counterpart controllability. Whether the simulated participant can be given a goal, a persona, a knowledge boundary and a patience budget, rather than only a prompt.
- Stopping conditions. Whether the harness supports goal-reached, give-up and turn-limit terminations as distinct outcomes, since conflating them corrupts every aggregate.
- Scenario reproducibility. Whether a scenario can be re-run with the same seed and persona and whether divergence across repeats is measured rather than hidden.
- Conversation-level scoring. Whether a scorer can be given the whole exchange, not only the last turn, and whether it can cite the turn where something went wrong.
- State inspection. Whether facts the system should have retained can be checked directly, rather than inferred from whether it happened to mention them.
- Depth economics. Cost per scenario grows with turn count and with context length at each turn, so a ten-turn suite is not ten times a one-turn suite but considerably more.
- Counterpart independence. Whether the simulator can run on a different model family from the system under test, to limit mutual accommodation.
Options
| Option | Best for | Watch out for |
|---|---|---|
| Model-driven user simulators with goal and persona specifications | Repeatable multi-turn coverage at a scale humans cannot provide | The simulator is a second system under evaluation; its cooperativeness determines what the suite can find |
| Multi-turn scenario support in evaluation frameworks (DeepEval conversational metrics, promptfoo conversation flows and comparable) | Getting scenario definitions and turn-aware scorers running without building a harness | Opinionated conversation shapes; simulators are often thin prompt wrappers with little goal discipline |
| Agent-and-user benchmark harnesses that pair a simulated customer with a policy document (τ-bench-style designs and comparable) | A methodology reference for combining conversational quality with a checkable end state | Domain-specific by construction; the value is the design, not the score |
| Recorded transcripts replayed for the next-turn prediction only | Cheap regression coverage on a single turn given a fixed history | Valid for exactly one turn; anything deeper measures a conversation the system would not have had |
| Human conversation studies with recruited participants | Ground truth about how real users behave when the system misunderstands them | Slow, expensive, and impossible to run per release; best used to calibrate simulators |
| Production conversation sampling with turn-aware scoring | Measuring real exchanges, including the length distribution and abandonment behaviour no simulation reproduces | No labels and no counterfactual; a long conversation may be engagement or may be failure |
| State-assertion harnesses over the conversation's retained context and task state | Checking directly whether a constraint stated at turn two survived to turn nine | Requires the system to expose its retained state; inferring retention from text is unreliable |