Multi-turn Evaluation — Patterns

Specify the counterpart by goal and stopping rule, not by script

Problem. A scripted participant cannot react, so the conversation it produces diverges from anything real the moment the system says something unexpected — which is exactly the case worth testing. A free-form simulated participant, on the other hand, wanders: it changes its mind, accepts answers it should reject, and eventually starts helping the system.

Approach. Define each scenario as a goal the counterpart is trying to achieve, a set of facts it knows and will reveal only when asked, a persona governing how it expresses itself, and explicit conditions under which it stops — goal satisfied, patience exhausted, or turn limit reached. Re-state the goal and the remaining facts to the simulator on every turn rather than relying on it to remember, and record which stopping condition fired as part of the result.

Trade-off. The counterpart becomes a system under test in its own right, needing its own validation against how real users behave, and its cooperativeness silently sets the ceiling on what the suite can discover. Withholding facts until asked also encodes an assumption about user behaviour that varies enormously between real populations.

Score the conversation and locate the turn

Problem. Averaging per-turn scores reports a conversation that went irrecoverably wrong at turn two as mostly fine, because turns three through ten were internally coherent continuations of a mistake. Scoring only the final turn is worse: it cannot distinguish an answer reached immediately from the same answer reached after eight turns of confusion.

Approach. Produce two outputs per scenario. The first is a verdict on the exchange as a whole, against the counterpart's stated goal: was it achieved, at what turn, and at what cost in turns. The second is the earliest turn at which the conversation went wrong, identified by a scorer reading the exchange in order and asked to name the first turn after which the goal became harder to reach. Aggregate the first for reporting and the second for triage.

Trade-off. First-failure identification is a judgement with real variance, and judges tend to name a turn where the problem became visible rather than where it originated. Running a judge over whole exchanges is also expensive and grows with conversation length, so this cannot be applied to every scenario at every depth.

Assert retention directly, at a stated distance

Problem. Whether the system still honours a constraint from turn two is usually tested by seeing if it mentions it later. That conflates retention with relevance — a constraint correctly held and not currently applicable looks identical to one that was forgotten.

Approach. Build scenarios where a constraint is stated early and then becomes decisive at a controlled distance: an allergy mentioned at turn one and a meal recommended at turn eight, a budget stated once and a proposal requested much later. The assertion is on the content of the decisive turn, and the distance is a parameter of the scenario, so the same constraint can be tested at three, eight and fifteen turns to find where retention breaks.

Trade-off. These scenarios are laborious to author and brittle — if the system reaches the decisive turn by a different route, or never reaches it, the assertion does not fire and the item has to be classified as inconclusive rather than passed. The distance parameter also interacts with everything else in the context, so the breaking point moves whenever anything upstream changes.

Repeat each scenario and report the spread

Problem. Two runs of the same scenario can diverge at the second turn and end in completely different places. A suite that runs each scenario once is comparing two systems on a single sample from a wide distribution, and a difference of several points may be nothing at all.

Approach. Run every scenario a fixed number of times with the counterpart's randomness varied, and report the distribution of outcomes rather than a point estimate: success rate across repeats, spread in turn count, and the fraction of repeats that reached the goal at all. Treat within-scenario variance as a published property of the suite, and require any claimed improvement to exceed it.

Trade-off. Repetition multiplies an already expensive suite by the repeat count, which usually forces either fewer scenarios or shallower ones — both of which reduce coverage of the late-turn failures that justify the capability. High variance is also uncomfortable to publish, and there is standing pressure to report the mean and omit the spread.