Jailbreak Detection — References

Why safety training fails in characterisable ways, how attempts are generated, what people actually try in the wild, and the published statements of what systems are supposed to refuse.

Why it is possible at all

• Jailbroken: How Does LLM Safety Training Fail? — Alexander Wei, Nika Haghtalab and Jacob Steinhardt, Neural Information Processing Systems, 2023. The two failure modes it names — competing objectives and mismatched generalisation — are the most useful framework available for predicting which new technique will work, and why a detector built on surface patterns will keep missing the next one. Cited by name.
• Zou, Wang, Carlini, Nasr, Kolter and Fredrikson, 2023, on universal and transferable adversarial attacks on aligned language models — cited here for the transferability result: a suffix optimised against one model works against others, which means an attacker does not need access to your deployment to search against something close enough to it. Cited by name.

How attempts are produced

• Jailbreaking Black Box Large Language Models in Twenty Queries — Patrick Chao, Alexander Robey, Edgar Dobriban, Hamed Hassani, George J. Pappas and Eric Wong, 2023. An attacker model refining its own prompts against a target within a small query budget, which is the shape that makes per-target search cheap and corpus-based detection obsolete. Cited by name.
• Great, Now Write an Article About That: The Crescendo Multi-Turn LLM Jailbreak Attack — Mark Russinovich, Ahmed Salem and Ronen Eldan, 2024. Escalation across turns from an entirely benign opening, and the clearest demonstration that a per-message judgement cannot see the attack. Cited by name.
• Many-shot Jailbreaking — Anthropic, 2024. Exploiting a long context by filling it with examples of the behaviour being solicited, notable because the technique scales with context length and therefore gets stronger with each model generation. Cited by name.

What people actually send

• "Do Anything Now": Characterizing and Evaluating In-The-Wild Jailbreak Prompts on Large Language Models — Xinyue Shen, Zeyuan Chen, Michael Backes, Yun Shen and Yang Zhang, ACM Conference on Computer and Communications Security, 2024. A longitudinal collection of prompts circulating in public communities, and evidence about how long individual templates remain effective. Cited by name.

Measurement

• HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusal — Mantas Mazeika and colleagues, International Conference on Machine Learning, 2024. A common harness and a defined attack success criterion, which matters mainly because success rates reported without a stated criterion are not comparable between two teams. Cited by name.
• Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learned — Ganguli and colleagues at Anthropic, 2022. The operational account of running human red teams at scale, including what attack rates look like over time and how attempts distribute across topics. Cited by name.
• Constitutional Classifiers: Defending against Universal Jailbreaks across Thousands of Hours of Red Teaming — Anthropic, 2025. Classifiers trained from an explicit written policy rather than from a labelled attack corpus, and a reported red-team result against a deployed configuration. Cited by name.

What the rules are

• The published usage policies of the major model providers — these are the operative statements of what a deployment is expected to refuse, and the reason a product's own detector must be measured against a written boundary rather than against intuition. Cited by name; they are revised frequently and should be read at their current source.

No verified reference is recorded for account-level attempt budgets, for uniform refusal surfaces as an anti-oracle measure, or for the false-positive burden of jailbreak enforcement on professional users. The published work measures attack success and does not measure what enforcement costs the people it catches by mistake.