Prompt Injection Detection

What it is

Runtime classification of whether a span of content in a request is attempting to redirect the system's behaviour, and the graded response once something is flagged. The material under examination is usually not the user's own message: it is a retrieved document, a tool result, a fetched page, an uploaded file, or a message from another agent. How context is built so that such material carries less authority — the marking, containment, and precedence of untrusted spans — is a separate capability; this one begins after that construction and asks whether a particular span is hostile.

The adversary is not the person asking

The hostile text here was written by someone who may never have touched the product, placed in a document or a system the product reads, and delivered by a user who is a victim rather than an attacker. That inverts the entire response. There is no account to rate-limit, the person in the session must still be served, and what they are told is a security notification rather than a refusal. The attacker also gets no feedback: they seeded a page weeks ago and cannot see whether it worked, which is the one structural advantage the defender has here and does not have against someone probing interactively from their own account.

Why it matters

Detection is the only component that can produce evidence an attack happened at all. Containment limits what a successful injection reaches but leaves no signal behind, so a compromised run is indistinguishable from an unhelpful one, and nobody learns that a supplier's document store, a shared inbox, or a public page has been seeded. It also supplies the trigger for graded degradation — read-only for the remainder of this turn, no outbound messages, drop this source — which is far cheaper than refusing and far stronger than hoping.

Key decisions

• What is scanned and when: every span at the moment it is admitted, the assembled context before the call, tool results as they return, or all three. Each sees a different form of the same text.
• The response ladder. Strip the span, discard the document, answer with a notice, reduce the tool set, or abandon the turn. A binary block makes the detector unusable at corpus scale.
• The base rate, and what a false positive costs. At one hostile document in a million, a detector that looked excellent on a benchmark removes legitimate content continuously.
• Whether the detector is itself a model reading attacker-controlled text, and what its output is permitted to be. A judgement expressed in free prose can be steered by the thing it is judging.
• How a flagged span is preserved for investigation without any path that puts it back into a context.
• Whether detection runs on the serving path at all, or asynchronously across the corpus, with the serving path relying on containment and on degradation triggered by cheaper signals.
• Who is told, and how quickly. The owner of a poisoned source is usually not the operator of the agent, and the useful output of detection is a report to them.

Failure modes

• Scanning runs on the final prompt, after a summarisation step has rewritten the injected instruction in the system's own voice, so the surface features the detector relies on are gone.
• The detector fires on material that legitimately discusses attacks — advisories, incident write-ups, policy text, training content — and the corpus loses exactly what a security team needs to read.
• A span is removed, the answer is produced without it, and nothing tells the user that the document they asked about was served with a hole in it.
• The payload is split across two retrieved chunks, each innocuous on its own, and the detector is only ever shown one chunk at a time.
• The instruction arrives as an encoded blob, a data URI, or a string in another language, which the model follows and the classifier scores as noise.
• The same detector both filters the serving path and labels its own retraining data, so it converges on the attacks it already catches and never learns the ones it misses.
• Flag rate is tracked per request rather than per source, so one poisoned repository firing across thousands of requests reads as diffuse background noise.
• Detection is wired to raise an alert and nothing is wired to reduce capability, so the compromised run finishes its tool calls while the alert sits in a queue.
• The detector is evaluated on a public corpus of attack strings and deployed against content that is specific to one organisation's documents, where none of those strings appear.

Related capabilities

• Jailbreak Detection — the same shape of problem with the account holder as the adversary
• Tool Guardrails — where a degraded capability set is actually enforced
• Data Leakage Prevention — the egress path a successful injection is usually aiming at
• Prompt Injection Defense — the containment this detection complements and cannot replace
• Context Provenance — the trust labelling that tells a scanner which spans are worth scanning

Applies: Production readiness · Design questions