Prompt A/B Testing

What it is

Running two or more prompt variants against live traffic at the same time, with assignment controlled, exposure recorded, and outcomes compared. It covers the unit that gets assigned, how assignment stays stable for that unit, what is logged so that a result can be attributed to a specific prompt revision, how long the comparison runs, and the conditions under which it is stopped early. The comparison happens on real requests and real outcomes; scoring variants against a fixed dataset before they reach traffic is an evaluation activity with its own methods.

Why it matters

Prompt changes are made constantly, justified by a handful of examples somebody tried, and shipped because the examples looked better. That procedure cannot detect a change that improves the cases a team thinks about and degrades the long tail it does not — which is the characteristic shape of a prompt regression. Concurrent comparison on live traffic is the only method that controls for the things that move on their own: the traffic mix, the model build, the season, and the product changes shipping the same week. It is also the only way to find out what a wording change does to a metric nobody associates with prompts, like refund rate or escalation volume.

Key decisions

• What is assigned — the request, the session, or the user. Per-request assignment maximises sample size and lets one user meet two different assistants mid-conversation; per-user assignment is stable and takes much longer to reach a conclusion.
• Whether the assignment is deterministic from an identifier so that a retried or replayed request lands in the same arm, which otherwise quietly contaminates both.
• What is measured. A prompt variant is a means to a product outcome, so the primary metric should be one the business already tracks, with model-side quantities treated as diagnostics.
• Which guardrail metrics stop an experiment regardless of the primary result: refusal rate, latency, cost per request, escalation, safety flags.
• How long the comparison runs, decided before it starts. Most prompt experiments are stopped the first day the dashboard looks favourable, which is how a measurement procedure becomes a random number generator.
• Whether everything that could differ between arms is held fixed — model build, inference settings, retrieval configuration, cache behaviour — because a prompt experiment that also changes the temperature has measured neither.
• How exposure is recorded. Without the assigned arm and the resolved prompt revision on the same record as the outcome, the analysis is a join that cannot be made.

Failure modes

• Assignment is per request, a user sees two different voices in one conversation, and the experiment measures the inconsistency rather than either variant.
• The variant is recorded but the resolved prompt revision is not. Someone edits a variant mid-experiment, the data spans two different prompts under one label, and the result is uninterpretable.
• One arm produces longer answers, which costs more and takes longer, and the measured preference tracks response latency rather than answer quality.
• The arms have different cache hit rates because one variant changed the stable prefix, so one arm is systematically faster and cheaper for reasons unrelated to its wording.
• A model build is updated mid-experiment. Both arms change at the same moment, the comparison is still valid in principle, and the absolute numbers before and after are silently incomparable.
• Significance is checked every morning and the experiment is stopped on the first favourable day, so the reported effect is an artefact of repeated looking.
• The variant wins on the primary metric and loses on one nobody was watching — more confident answers, fewer clarifying questions, and a higher downstream correction rate three weeks later.
• Traffic is routed to different models by a separate routing policy correlated with request difficulty, so the two arms received different distributions of hard requests.
• The experiment ends, the winning variant is promoted, and nothing records which revision won or why, so the same question is re-litigated six months later.

Related capabilities

• Prompt Registry and Versioning — supplies the stable revision identity each arm is attributed to
• Few-shot Examples — a common subject of comparison, and one that changes cost as well as quality
• System and Developer Instructions — the text whose proposed edits this capability adjudicates
• Dynamic Prompt Assembly — must hold every non-tested input identical across arms
• Context Window Management — a longer variant changes occupancy, which is a confound and a cost

Applies: Production readiness · Design questions