Online Evaluation — References

Sources on interpreting behavioural signals, on the instrumentation a live quality join depends on, and on why observed movements usually are not what they appear to be.

Reading user behaviour as a quality signal

• The information-retrieval literature on implicit feedback from click logs — established well before language models and largely transferable — is the standard treatment of why observed user actions are biased by position, presentation and examination rather than by quality alone. Described here rather than attributed to one title, because the central results are spread across several papers by overlapping author groups.
• Holistic Evaluation of Language Models (Liang et al., 2022), arXiv:2211.09110 — relevant here for the insistence that a system is characterised by several metrics reported together, which is the discipline a live dashboard most often lacks.

Instrumentation

• OpenTelemetry semantic conventions — the generative-AI section defines where model identity, request parameters and token counts belong on a span, which is what makes a quality signal segmentable by build after the fact.

Interpreting a movement

• The experimentation literature from large-scale web platforms, on trustworthy online measurement and the many ways an apparent improvement turns out to be instrumentation, seasonality or a changed population, is directly applicable and well documented in practitioner books and conference tutorials. No single reference is recorded here, since the material most relevant to this capability — observational monitoring rather than controlled experiment — is scattered across that body of work.

No verified reference is recorded for stratified sampling rates in production language-model scoring, for asynchronous judge pipelines, or for the retention design that live transcript scoring requires.