Multi-turn Evaluation

What it is

Measuring a conversation rather than a response. The object under test is an exchange in which each thing the system says changes what the other participant says next, so the input to turn six does not exist until turns one through five have been produced. Quality here is a property of the whole exchange — whether context established early was still honoured late, whether a correction was absorbed, whether the system arrived somewhere useful before the user gave up.

Why a recorded conversation cannot be replayed

A stored transcript is a valid test case for exactly one turn. Feed the system the first user message and it produces something different from what was recorded; from that point the recorded user messages are replies to statements that were never made, and every subsequent turn is being evaluated on an input that makes no sense. The usual workaround — forcing the recorded assistant turns back into the history and only scoring the final one — measures a system that never existed, because the real product would have reached turn six from its own turn five. Getting past this requires a participant that responds to what was actually said, which means either a person in the loop or a simulated user, and both introduce a second system whose behaviour is now part of the measurement.

Why it matters

Most of the production failures in conversational products are failures of the fifth turn, not the first. Constraints stated early are dropped once the window fills, a correction is acknowledged and then ignored, a misunderstanding at turn two is elaborated confidently for six turns, and a question the user already answered gets asked again. Single-response evaluation is blind to all of it by construction — every one of those turns, assessed alone against its immediate input, looks perfectly reasonable. The compounding is the defect, and it only exists at the level of the exchange.

Key decisions

• Who plays the user. A scripted participant is reproducible and will not react; a simulated one reacts and brings its own model's quirks; a real one is the ground truth and cannot be run on every release.
• How a simulated user is specified — a persona, a goal, a stopping condition, a patience budget — since without a stopping rule the simulation continues until something arbitrary cuts it off.
• Whether the score belongs to the conversation, to each turn, or to both. A turn-level score localises a defect; a conversation-level score is the thing users actually experience.
• How a defect is attributed across turns, given that the turn where the damage becomes visible is usually several after the turn that caused it.
• What state is inspected besides the text — retained facts, open commitments, accumulated context, any task state the conversation is driving.
• How many turns deep the evaluation goes, and on what basis, since cost scales with depth and the interesting failures live past the point most suites stop.
• Whether the conversation's variance is measured by running the same scenario several times, because two runs of one scenario can diverge completely from the second turn.

Failure modes

• Recorded transcripts are replayed with the original assistant turns forced into the history, and the suite measures a conversation that could never occur.
• The simulated user is cooperative, accepts the first answer and never pushes back, so the suite never reaches the situations where real conversations fail.
• The simulated user is played by the same model as the system, and the pair converge on a mutual style that nothing in production resembles.
• Every scenario stops at three turns because that was the default, and the entire class of constraint-forgetting failures sits at turn seven.
• Scoring is per turn and averaged, so a conversation that went permanently wrong at turn two is reported as eighty percent good because the subsequent turns were internally coherent.
• Only the final turn is scored, and a system that wandered for ten turns before answering correctly outscores one that answered at turn two.
• The user simulator drifts from its stated goal and starts agreeing with the system, which is scored as a successful conversation.
• Each scenario is run once, the run-to-run divergence is larger than the difference between two candidate systems, and the comparison is decided by sampling noise.
• A conversation that correctly ends — the goal was met, nothing more was needed — is scored as an early termination failure, and the system is tuned to keep talking.

Related capabilities

• Agent Trajectory Evaluation — the same sequential attribution problem where the other party is an environment rather than a person
• Task Completion Evaluation — what a long exchange was ultimately for
• Human Evaluation — the only participant that reliably behaves like a real user
• LLM-as-a-Judge — the scorer asked whether a commitment made early survived to the end
• Golden Datasets — holds scenarios, personas and goals rather than single inputs

Applies: Production readiness · Design questions