Adversarial Testing

What it is

Deliberately constructing inputs intended to make the system behave badly, and recording what worked. The posture is generative rather than comparative: there is no fixed set to score against, because the activity is the search for an input that is not in any set yet. Its product is a collection of working attacks and an account of how hard they were to find — which classes of manipulation succeed, at what effort, against which part of the system. Measuring compliance against a written standard is a different activity that consumes some of this one's output.

Why it matters

Every other instrument in the evaluation stack measures behaviour on inputs somebody expected. Real failures in deployed systems are overwhelmingly caused by inputs nobody expected, produced by people who were actively looking for them — users probing for entertainment, customers seeking an advantage, and attackers with a target. There is also an asymmetry that makes passive measurement insufficient: a defect needs to be found once to be exploited indefinitely, while a test suite has to anticipate it in advance. Adversarial work is the only activity in the stack that models the person on the other side as intelligent and persistent.

Key decisions

• Who does the attacking. Internal engineers know the architecture and share its blind spots; external specialists bring unfamiliar technique and no context; automated generators produce volume and little creativity.
• What the attacker is assumed to know and to control — the prompt, the tool surface, documents that will be retrieved, the content of a webpage the system will read — since the threat model determines which attacks are in scope.
• Whether the target is the model's behaviour, the surrounding application, or the boundary between them. The most damaging findings are usually at the boundary, where a manipulated output becomes an action.
• How a finding is generalised. Fixing the exact string that worked is almost always worthless; identifying the class and testing the class is the whole value of the exercise.
• Whether successful attacks are retained as a permanent suite, and how to account for the fact that a retained attack is one the system has since been specifically hardened against.
• How effort is recorded, since the useful measurement is not whether an attack exists but how much work it took to find one — and that number only means something if it is tracked over time.
• What happens to the findings: an inventory, a blocked release, a fix, or a disclosure path for attacks found by people outside the team.

Failure modes

• A fixed library of known attack strings is run every release, the system passes, and a defect is read as a safety property when it is a record of what was already patched.
• The exact prompts that succeeded get blocked verbatim, a trivial rephrasing works, and the team believes the class is closed.
• Attacks are written only by the people who built the system, and every one of them assumes the interface works the way the architecture diagram says.
• Only the model is probed, while the actual exposure is a tool the model can invoke with arguments an attacker can influence.
• Content that arrives through retrieval or a fetched page is never treated as attacker controlled, so the entire indirect delivery path is out of scope by accident.
• Success is reported as a rate against a fixed set, which falls as the set is patched and approaches zero while the system is no more robust than before.
• A finding is fixed without being turned into a test, and it returns three releases later when the prompt is rewritten.
• The exercise runs once before launch, the system changes continuously afterwards, and nothing re-probes it.
• Attacks are generated by a model from the same family as the one under test, and the generator will not produce the attacks it is itself trained to refuse.

Related capabilities

• Safety Evaluation — measures the system against a written standard, using findings this activity produces
• Golden Datasets — where a generalised attack class becomes permanent coverage
• Regression Evaluation — confirms a hardening change did not quietly undo an earlier one
• Human Evaluation — adjudicates whether an attacked output was actually harmful
• Evaluation Gates — where an unresolved finding becomes a release blocker

Applies: Production readiness · Design questions