Data Leakage Prevention

What it is

Stopping confidential material from reaching a party not entitled to it through anything the system emits — the answer, the arguments of an outbound call, a log line, an attachment, a generated link. The judgement is about a disclosure rather than about a string: it combines what the content is, which store it came from, and who is about to receive it.

Classifying the text is not classifying the disclosure

Whether a span is personal data is a property of the span, and a detector can decide it by reading. Whether this paragraph may be shown to this recipient is not a property of the paragraph at all: the same text is routine for one audience and a reportable incident for another, and nothing in it says which. That is why sensitivity detectors are necessary here and never sufficient — the missing half is the entitlement of the receiver, which has to travel with the request, and the record of which source the content was drawn from. Deciding who may read which store is a separate control that runs before anything is retrieved; this one assumes those decisions exist and watches what crosses the boundary anyway, because a model that read three stores legitimately can write one answer that no single entitlement covers.

Why it matters

A model is a mixing device. It reads from several sources with different entitlements and emits one piece of text carrying none of their labels, and every component downstream treats that text as ordinary output. The characteristic breach in these systems is not a stolen credential; it is a correct, helpful, well-formatted answer assembled from a document the asker was never cleared for, or an agent persuaded to place a customer list in the body of an email. The leak is produced by the system working exactly as designed, which is why nothing in the error budget or the security monitoring will show it.

Key decisions

• Whether sensitivity is inherited from the source or inferred from the content, and what happens to derived text. A summary carries the secret and none of the labelling.
• Which egress paths are in scope. Covering the response and not the outbound call covers the smaller half, because an agent that can send is an agent that can send anywhere.
• The granularity of enforcement — the whole response, the paragraph, or the individual claim — and whether partial withholding is something the product can present coherently.
• Whether the check can compare what is being emitted against the entitlement of the recipient, which requires that entitlement to be present at the point of the check and not merely at retrieval time.
• What happens when a legitimate answer depends on inadmissible material: refuse, answer partially, or explain the gap in terms that themselves confirm the material exists.
• Whether production content may flow into evaluation sets, prompt caches, fine-tuning corpora, and vendor-side retention — disclosures with no recipient and no moment anyone would call a breach.
• Which exfiltration shapes are detectable at all: encoded strings, long opaque identifiers, payloads hidden in URL paths, content split across several outputs.

Failure modes

• Retrieval honours the asker's entitlements and the conversation history does not, so material fetched for one participant in a shared session is quoted back in another's turn.
• A source document is labelled and its summary is not, so a second retrieval pass over summaries serves restricted content as though it were public.
• The egress check inspects the assistant's message while the agent has already sent the content as an argument to a search tool, a webhook, or an outbound email.
• A tenant filter is enforced in the primary store and omitted from the vector index, so a nearest-neighbour query crosses a boundary the relational query could not.
• Content is encoded, or spread across several responses, and a per-response check sees nothing matching any signature.
• A generated link carries retrieved text in a query parameter, and the act of fetching a preview delivers it to a third party.
• Traces, prompt logs, and error payloads are exported to an observability vendor unredacted, so the leak is in the tooling rather than in the product.
• A clean verdict is cached against the content and later served to a different recipient, for whom the judgement was never made.
• The agent is denied a document and answers from the retrieval snippet that the denial message included.

Related capabilities

• PII Detection and Masking — the intrinsic content judgement this relational one builds on
• Tool Guardrails — the check on outbound arguments, which is the other half of egress
• Prompt Injection Detection — the attack that most often drives an agent to exfiltrate
• Output Guardrails — the checkpoint where a response-side check is mounted
• Credential and Secret Management — custody of secrets, as distinct from inspecting text for them

Applies: Production readiness · Design questions