Online Evaluation

What it is

Judging a system from what happens when real people use it, where no expected answer exists for any individual request. Quality has to be inferred from what users did next — accepted the output, edited it, asked again, abandoned the session, escalated to a person, or came back a week later — combined with scorers run over a sample of recorded traffic. The population is not chosen, the inputs are not repeatable, and the signal arrives late and unevenly.

Where this stops and controlled comparison begins

Observing one live system tells you its current state: what fraction of sessions end in an escalation, which intent classes draw the most retries, whether yesterday looked like last Tuesday. It cannot attribute any of that to a decision, because everything moved at once — the traffic mix, the season, the model build, the three other things that shipped that week. Establishing that one variant caused an outcome requires randomised concurrent assignment, which is a separate discipline with its own arithmetic about assignment units, exposure records and stopping rules. This capability supplies the instrumentation and the outcome definitions that such a comparison consumes, and takes over again once the experiment ends and the winner has to be watched indefinitely.

Why it matters

Every offline number is a claim about a sample somebody chose, and the only test of that claim is what the system does on traffic nobody curated. Live measurement is where the requests nobody imagined show up: the ones in the wrong language, the ones pasted from a spreadsheet, the ones where the user wanted something the product does not do. It is also the only place certain quality properties exist at all — whether an answer was useful is partly a fact about the person who received it, and no fixed dataset contains that person.

Key decisions

• Which behavioural signals are treated as quality evidence, and what each one actually encodes. A copy action is a strong positive and a rare one; an absence of complaint is not a positive at all; an explicit thumbs-down is informative and is produced by an unrepresentative minority.
• Where the outcome is observed. Signals inside the assistant are immediate and shallow; signals in the surrounding product — ticket reopened, refund issued, code reverted — are the ones the business cares about and arrive days later.
• What fraction of traffic is scored by automated judges, since running a scorer on every request doubles the cost of the product and running it on one percent cannot see a rare failure class.
• How the sample is drawn. Scoring only completed sessions removes exactly the traffic that went wrong, and scoring only flagged sessions measures the flagging rule.
• Whether per-request records carry enough identity — build, prompt revision, retrieval configuration, routing decision — to segment a movement later, since that cannot be added retroactively to traffic already served.
• How the measurement respects the obligations attached to production content: what may be retained, for how long, who may read a transcript, and whether a scorer may see raw user text at all.

Failure modes

• Satisfaction is measured by a thumbs widget answered by a fraction of a percent of sessions, and that fraction is overwhelmingly people who were annoyed.
• A metric improves because the product got slower and impatient users left, so the remaining population is more tolerant and nothing about the answers changed.
• Automated scoring runs on a sample drawn after sessions that errored were filtered out, so the quality dashboard is computed exclusively over requests that worked.
• The headline signal is average response rating, which is flat forever because it is dominated by easy requests, while the hard intent class that matters degrades invisibly underneath.
• An outcome is defined as "user did not retry within five minutes", and a latency increase pushes retries past the window, which registers as a quality improvement.
• The build identifier is not recorded on requests, a regression is noticed a week later, and nobody can say which of four releases it started in.
• Session-level signals are attributed to the last turn, so a failure at turn two is recorded against a perfectly good answer at turn nine.
• A scorer is changed to a different model mid-quarter, every historical number shifts, and the break is read as a product regression.

Related capabilities

• Offline Evaluation — the pre-release measurement whose blind spots this one exists to cover
• Regression Evaluation — deciding whether an observed movement is real
• LLM-as-a-Judge — the scorer most often run over sampled live traffic
• Confidence Calibration — turns a live confidence signal into something an escalation rule can use
• Golden Datasets — where traffic discovered here should end up as permanent coverage

Applies: Production readiness · Design questions