Jailbreak Detection

What it is

Recognising that the person in the session is working to get the system outside the behaviour its operator defined, and deciding what the product does about that person as well as about the message in front of it. The signal is rarely a single sentence: it is a trajectory — the framings tried, the reformulations that follow each refusal, the escalation across turns, and the shape of an account over days. Content that is simply disallowed, and third-party text trying to hijack a run, are judged by other checks; this one is about a user's own sustained effort to defeat the rules.

Why it matters

The adversary here has everything a researcher has: unlimited attempts, a visible response to each one, and no cost for failing. Any defence resting on refusing individual messages is being hill-climbed in real time by the person it is refusing, and the refusal itself is the gradient. Moving the response from the message to the session and the account is the only way to make an attacker's budget finite. It is also the only route to an honest answer about whether published safety behaviour is holding, because the number that matters is not how often the model refuses but how often someone determined eventually gets through.

Key decisions

• The unit of judgement: the message, the conversation, the session, the account, or the device and network behind it. Anything narrower than the conversation misses the dominant attack shape.
• What happens on a finding beyond refusing — slowing responses, withdrawing capabilities, requiring verification, queueing for review, suspending. Each carries a different false-positive cost and a different appeals burden.
• How much the response reveals. A refusal that varies informatively with the attempt is a gradient; a refusal that is identical for every cause tells the attacker nothing and tells honest users nothing either.
• How legitimate lookalikes stay usable: security research, fiction, clinical and harm-reduction questions, and your own red team. Whether an exemption exists, who grants it, and what it costs to get one.
• Whether the output is used for enforcement, for measurement, or both, given that a signal stops being a measurement the moment attackers begin adapting to the enforcement it drives.
• What the maintained evaluation is. A frozen corpus of published attempts decays into a regression test for last year's techniques within months.
• Who adjudicates a contested enforcement action, how quickly, and whether the person affected is told enough to contest it at all.

Failure modes

• The detector is trained on published attempts, and the attacks that matter are generated per target by an automated search producing strings that appear in no corpus.
• Judgement is per message, so an escalation that opens with a plausible premise and moves one step per turn never presents a message worth flagging.
• An account is suspended on a score with no human review path, and the first appeals arrive from exactly the professional population the product was built for.
• The system refuses and still answers partially, and the fragments are harvested across twenty sessions into the complete output nobody ever returned in one piece.
• The detector and the model's own refusal behaviour are owned by different teams, so the measured success rate reflects whichever is weaker and nobody owns the number.
• Attempts are counted per conversation, and a new conversation is free.
• Role-play and hypothetical framings are blocked wholesale, and a product whose main use is fiction loses its core workflow to a control aimed at someone else.
• A detected attempt is answered more slowly than an ordinary request, which confirms detection and tells the attacker precisely which variant to keep developing.
• Enforcement is applied at the account level while the product allows anonymous trials, so the cost of a fresh identity is lower than the cost of a better attack.

Related capabilities

• Prompt Injection Detection — the same problem where the hostile text was written by a third party
• Content Moderation — judges what was produced, where this judges what was attempted
• Input Guardrails — the checkpoint this judgement is usually mounted on
• Policy Enforcement — defines the boundary an attempt is trying to cross
• Fail-closed Behavior — what happens to a session when this detector is unavailable

Applies: Production readiness · Design questions