Online Evaluation — Patterns

Define the outcome in the product, not in the assistant

Problem. Signals available inside the generation loop — rating widgets, regenerate clicks, answer length — are cheap and weakly connected to whether anything useful happened. Teams optimise them for a quarter and discover that the business metric never moved.

Approach. Pick a small number of outcomes the organisation already tracks and would act on independently of any model: ticket resolved without escalation, draft sent without substantive edit, code merged without revert, order completed. Instrument the assistant interaction so that it carries a correlation identifier into those systems, and define quality as the rate of the downstream outcome conditioned on an interaction having occurred. Model-side quantities stay in the dashboard as diagnostics for explaining a movement, never as the movement itself.

Trade-off. Downstream outcomes are confounded by everything else in the product and arrive too late to catch a bad release within the hour. They also require a join across systems owned by other teams, which is an organisational dependency rather than a technical one, and it is the usual reason this pattern is proposed and not built.

Stratify the scored sample by what you need to see

Problem. A uniform one-percent sample is dominated by the easy majority of traffic. The intent class that fails, the tenant with the unusual document format, the non-English requests — each contributes a handful of scored items per week, far too few to detect a change in.

Approach. Partition traffic into strata that matter — intent class, tenant tier, language, whether a tool was called, whether retrieval returned nothing — and set a sampling rate per stratum sized to the detection sensitivity each one needs. Record the stratum and its sampling weight on every scored item, and reweight when computing any population-level figure, so the oversampled rare classes do not distort the headline number.

Trade-off. Weighted aggregates are easy to get wrong and hard to debug, and a stratum definition that shifts mid-quarter silently invalidates the reweighting. The strata also have to be chosen in advance, which means the failure class nobody anticipated is still under-sampled.

Score asynchronously from the recorded trace

Problem. Running a judge inline adds its latency and its failure modes to every request it touches, and the pressure to keep the request fast ends with the scorer being skipped exactly when the system is under stress — which is when its output matters most.

Approach. Make the serving path emit a complete trace and nothing else: inputs, resolved context, intermediate calls, final output, configuration identities. Scoring runs as a separate consumer over those traces, at whatever rate and cost is affordable, and writes scores back keyed by the trace identifier. The scorer can then be changed, re-run over history, or run twice with different judges, none of which touches the product.

Trade-off. The trace must contain everything the scorer needs, and discovering a missing field means waiting for new traffic rather than re-running. Retaining full traces also maximises exactly the data that privacy and retention policy wants minimised, which forces a redaction design before this pattern can ship.

Keep a continuously refreshed traffic mirror for diagnosis

Problem. When a live metric moves, the question is immediately "on what kind of request", and the aggregate cannot answer it. Reconstructing the affected population after the fact means querying raw traces under time pressure with a half-remembered filter.

Approach. Maintain a rolling sample of recent requests, stratified and refreshed on a fixed cadence, stored with their full resolved context and their observed outcomes. It serves as the standing answer to "what does traffic look like now", as the source from which new permanent test cases are promoted, and as the population a suspected regression can be replayed against immediately.

Trade-off. A mirror of production content inherits every obligation attached to the original, including deletion requests, and a rolling window that silently retains a user's text past its retention period is a compliance incident rather than an evaluation asset.