Evaluation and Quality Engineering
Online Evaluation — Stack
Online evaluation › Stack
Technology landscape as of 2026-10-10. Knowledge cutoff May 2026.
Selection criteria
The components here capture what happened, join it to what happened next, and score a slice of it. The hard requirement is the join: a quality signal that cannot be connected to the request that caused it is decoration.
- Correlation identity. Whether every emitted record — request, generation, tool call, user action, downstream business event — carries a shared identifier that survives queues, retries and service boundaries.
- Late-arriving outcome support. Whether the storage layer can attach a fact observed three days later to a request served on Monday, without rewriting history or double-counting.
- Sampling control. Whether the sample rate can be set per route, per tenant, per intent class and per outcome, rather than as one global percentage that under-samples the rare failures you need.
- Payload handling. Whether prompts and responses can be redacted, hashed, truncated or excluded by field before leaving the process, since production text carries retention and residency obligations.
- Asynchronous scoring. Whether scorers run off the serving path entirely, so that a slow or failing judge degrades the dashboard rather than the product.
- Cardinality tolerance. Whether segmenting by prompt revision, model build and tenant is affordable, because those are exactly the dimensions a live investigation needs.
- Retention and cost shape. How long raw traces are kept at full fidelity versus aggregated, since trace volume scales with traffic and quality investigations need the raw form.
Options
| Option | Best for | Watch out for |
|---|---|---|
| LLM-specific tracing and evaluation platforms (LangSmith, Langfuse, Arize Phoenix, Braintrust, Weights & Biases Weave and similar) | Prompt, response and score inspection with the generation semantics already modelled | Pricing scales with trace volume; the quality record lives apart from the rest of the product's telemetry |
| General observability backends fed through OpenTelemetry generative-AI conventions (Grafana, Datadog, Honeycomb and comparable) | Keeping model quality signals in the same system as latency, errors and infrastructure | Span attribute limits and sampling defaults truncate prompts and responses; no notion of a score |
| Product analytics platforms (Amplitude, Mixpanel, PostHog and similar) | Connecting an assistant interaction to the downstream user behaviour that defines the outcome | Event models assume user-initiated actions; a generation is awkward to represent and expensive at per-token cardinality |
| Warehouse plus scheduled transformations (BigQuery, Snowflake, ClickHouse with dbt-style modelling) | Late-arriving outcomes, long retention, and joins to business facts that only exist in the warehouse | Latency measured in hours; nothing is interactive during an incident |
| Streaming pipelines for near-real-time scoring (Kafka or Kinesis with a stream processor) | High traffic where a rare failure class needs to be detected within minutes | Substantial operational surface, and exactly-once semantics that quality aggregates quietly depend on |
| In-product feedback capture (rating widgets, inline edit capture, regenerate and copy instrumentation) | Cheap direct signal and the implicit signals that are far more representative than explicit ones | Explicit ratings are self-selected and sparse; implicit signals need careful definition before they mean anything |
| Session replay and transcript review tooling | Qualitative investigation of a failure class a metric has pointed at | Access to raw conversations is a privacy decision, not a tooling decision, and needs controls before it is enabled |