Adversarial Testing — Patterns

Measure effort to first success, not pass rate

Problem. A rate against a stored attack set looks like a robustness measurement and is not. It falls as the system is patched against exactly those attacks, reaches zero, and keeps reporting zero while a fresh attacker still succeeds in twenty minutes.

Approach. Make the unit of measurement the search: give a tester or an automated attacker a defined objective and a budget, and record how many attempts and how much time elapsed before the first success, alongside what finally worked. Track the distribution of that number across objectives and over time. A rising effort-to-success is the only evidence of hardening that survives patching, and a sudden fall after a release is a signal nothing else produces.

Trade-off. The number is noisy and heavily dependent on who is attacking, so comparisons across testers or across quarters with different personnel mean little. It is also expensive to produce, because each measurement requires an actual search rather than a replay.

Generalise every finding into a class before fixing it

Problem. A working attack arrives as a specific string. The fastest fix is to block that string, and it is almost always worthless — the next phrasing, the same request in another language, the same structure with different content, all still work. Teams accumulate a long list of blocked specifics and no improvement in robustness.

Approach. For each finding, write down the mechanism: what property of the system did it exploit, what is the general shape of inputs that exploit it, and what is the smallest change that would stop the class rather than the instance. Generate a family of variants around the mechanism — translations, encodings, indirection through a retrieved document, the same manipulation split across turns — and treat the fix as unverified until the family fails.

Trade-off. Generalisation is slow, requires someone who understands the system deeply, and sometimes produces a class so broad that no fix short of removing a feature addresses it. The variant family also becomes a maintenance burden, and its members age at different rates.

Attack through every channel the product reads from

Problem. Exercises concentrate on the input box because that is where a tester types. The exposures that cause incidents are usually elsewhere: a document in the corpus, a tool's response, a fetched page, a file a user uploaded, a field in a record the system summarises. Every one of those is text that reaches the model, and in many products some of it is attacker controlled.

Approach. Enumerate every path by which text reaches the model, mark each as trusted or attacker-influenced under the threat model, and require the exercise to deliver attacks through each influenced path rather than only through the direct one. Planting a payload in a document the system will retrieve, or in a tool response it will act on, is frequently far more effective than anything typed directly, and is the path least likely to be covered.

Trade-off. Testing indirect paths requires write access to the corpus or control of a tool backend, which is awkward to arrange and risky in anything resembling production. The enumeration also dates immediately — every new integration adds a path, and the list is maintained by nobody in particular.

Keep the attack corpus, and know what it has become

Problem. Retained attacks are the obvious asset of this work and the easiest thing to misread. After a few releases, the system has been hardened specifically against them, so the suite passes completely and conveys an assurance it cannot support.

Approach. Keep the corpus for regression purposes and label it as such. Record for each attack the date found, the mechanism, whether a fix was shipped against it specifically, and the date it last succeeded. Report it as a regression suite — these previously worked and no longer do — and never as a coverage or robustness figure. Pair every release-time run of the corpus with a fresh search whose findings are reported separately.

Trade-off. Maintaining the corpus costs effort for a suite that by design approaches a perfect score and stays there, and the separation between regression and discovery is one that summaries erase. Retaining working attacks also creates a sensitive artefact that needs access control of its own.