PII Detection and Masking — Patterns

Structure first, statistics second

Problem. A single detector asked to find everything has one operating point, and the classes it covers have wildly different error profiles. Account numbers either validate or they do not; names are a judgement. Tuning one threshold for both means accepting either missed names or mangled invoices.

Approach. Run the deterministic detectors first and let them claim their spans with certainty: checksum-validated identifiers, structurally verified contact details, bounded formats. Remove those spans from consideration, then run the statistical detector over what remains, with its own threshold chosen for the classes only it can find. Report precision and recall per class rather than per document, because a document-level figure is dominated by whichever class is most common.

Trade-off. Two stages mean two places a span can be classified, and overlapping claims need an arbitration rule that will occasionally pick the wrong one — a long number inside a sentence can be an account number or a quantity. The staged design also makes the pipeline harder to reason about when someone asks why a particular value survived.

Stable pseudonyms from a scoped key

Problem. Replacing every occurrence with an identical marker destroys the text: a conversation between three people becomes a conversation between three instances of the same token. Replacing each occurrence with a distinct marker is worse, because it breaks coreference and the reader can no longer tell whether two mentions refer to one person.

Approach. Derive the placeholder deterministically from the value and a key scoped to the narrowest useful boundary — usually the document or the conversation — so the same person is the same pseudonym inside it and an unrelated pseudonym outside it. Keep the pseudonym type-shaped, so a name is replaced by something name-shaped and a date by something date-shaped, which preserves the grammatical and structural cues a reader and a parser both rely on.

Trade-off. Scope is a direct trade between usefulness and linkage: widening it lets you join records about the same person, which is sometimes the point and is also precisely the capability pseudonymisation was supposed to remove. A deterministic derivation is also vulnerable to a dictionary attack by anyone holding the key and a list of likely values.

Transform at the boundary data crosses, not at the surface it shows

Problem. Masking implemented in the rendering layer is the easiest place to put it and the least useful. By the time text is being displayed it has already been written to the request log, the trace exporter, the prompt cache, the evaluation store, and the provider's retention window.

Approach. Enumerate the boundaries where data leaves the system's own control — the inference provider, the trace backend, the analytics warehouse, the export path, the evaluation corpus — and place the transformation at each, with rules suited to what that boundary needs. The display layer is then one boundary among several and usually the least strict, because the person looking at the screen is often entitled to see the real value.

Trade-off. Several transformation points mean several configurations to keep in step, and the one that gets forgotten is the one added last. Masking before the inference provider also removes information the model needed, so this pattern forces an explicit decision about which boundaries can afford degraded text and which cannot.

Seeded records as the recall measurement

Problem. Recall cannot be measured without labelled data, and the labelled data would be a corpus of real personal information retained specifically so it can be looked at. The measurement you need is the one your policy forbids you to construct.

Approach. Manufacture it. Generate synthetic records that match the distribution of your real data — including the locales, scripts, and malformed variants that actually occur — inject them into the paths under test, and measure how many survive. Keep the synthetic corpus in version control, add to it every time a real miss is discovered and characterised, and treat a drop in its pass rate as a release blocker.

Trade-off. Synthetic data is systematically easier than real data: it is well-formed, correctly encoded, and free of the truncations, transcription errors, and mixed-script oddities that cause real misses. The measurement is therefore an upper bound, and treating it as the true recall is how teams come to believe a detector is better than it is.