Evaluation and Quality Engineering
Adversarial Testing — Stack
Adversarial testing › Stack
Technology landscape as of 2026-10-10. Knowledge cutoff May 2026.
Selection criteria
The options are ways of producing candidate attacks and of determining whether one worked. The second is harder than it sounds: deciding that an output constitutes a successful attack is itself a judgement.
- Threat-model coverage. Whether the mechanism can deliver an attack through every channel the product exposes — direct input, retrieved documents, tool results, uploaded files, linked pages — rather than only through the chat box.
- Success determination. How the tool decides an attack succeeded, and whether that decision is itself reliable enough to report a rate from.
- Generalisation support. Whether findings are organised by class and mutated systematically, or accumulate as a flat list of strings.
- Novelty over time. Whether the mechanism keeps producing attacks the system has not seen, or converges on a fixed repertoire within a few runs.
- Full-system reach. Whether the target is the deployed application with its real tools and retrieval, or a model endpoint in isolation.
- Effort instrumentation. Whether the number of attempts to first success is recorded, since that is the measurement that survives patching.
- Containment. Whether attacks can run against a system with real capabilities without actually causing the effects they are probing for.
- Reporting obligations. Whether findings can be handled under the disclosure and severity process the organisation already has.
Options
| Option | Best for | Watch out for |
|---|---|---|
| Expert human red-teaming, internal or contracted | Genuine novelty, boundary attacks between the model and the application, and anything requiring domain context | Expensive, intermittent, and quality depends entirely on the individuals; coverage is unquantifiable |
| Open-source adversarial testing frameworks (Garak, PyRIT, promptfoo red-team modules and comparable) | Repeatable coverage of known attack families with a scoring harness already built | Known families converge quickly; what they measure is resistance to published techniques, not robustness |
| Automated attacker models driving a conversation against the target | Volume, multi-turn attacks, and probing that adapts to what the target just said | The attacker's refusals cap what it will generate; attacker and target from the same family share blind spots |
| Published attack taxonomies and catalogues (OWASP's language-model project, MITRE ATLAS and comparable) | Structuring an exercise so that whole categories are not forgotten | A catalogue is a checklist, not a search; passing every listed item proves nothing about unlisted ones |
| Crowdsourced and bug-bounty style programmes | Scale, unfamiliar perspectives, and sustained probing after launch | Needs triage capacity, a severity rubric and a payment model; low-quality submissions dominate without them |
| Fuzzing and mutation over known successful attacks | Turning one working attack into a tested class cheaply | Mutations stay close to the original; this measures the patch's brittleness, not the space of attacks |
| Isolated replica environments with instrumented tool surfaces | Probing systems with real capabilities, where a successful attack must not have real effects | Replica drift makes a finding unreproducible in production, and the replica's safety wiring can itself mask the defect |