Guardrails and AI Safety
PII Detection and Masking — Stack
Pii detection and masking › Stack
Technology landscape as of 2026-10-10. Knowledge cutoff May 2026.
Selection criteria
You are buying recall on entity classes you must cover, in the languages you serve, plus a transformation you can live with for as long as the data exists.
- Entity coverage against your obligations. Not the vendor's list but the intersection of their list with the classes your regulators, contracts, and customers name.
- Language and script performance, measured separately. An averaged figure conceals that recall on one script is a fraction of recall on another.
- Transformation modes. Whether the component only removes, or can also tokenise, hash, generalise, and produce a format-preserving substitute that downstream parsers still accept.
- Placeholder consistency. Whether the same input value yields the same placeholder within a defined scope, and whether that scope is configurable.
- Reversibility custody. Where the mapping is held, who can resolve it, and whether resolution is itself audited.
- Confidence exposure. Whether per-span scores are available so the uncertain band can be routed somewhere other than straight to replacement.
- Data residency of the detector. Whether protecting data from one processor means sending it to another.
- Throughput. Cost and latency over the volumes a trace pipeline produces, which is a different order of magnitude from the request path.
Options
| Option | Best for | Watch out for |
|---|---|---|
| Pattern rules with structural validation (checksum-verified account and card numbers, national identifier formats, bounded date and contact patterns) | Identifiers with a defined structure, where a validator makes precision close to exact at negligible cost | Says nothing about names, addresses, or free-form description; patterns without checksums generate constant false positives in code and logs |
| Named-entity recognition pipelines (Presidio with spaCy or transformer recognisers, stand-alone NER models) | Names, organisations, locations and other entities no pattern describes | Recall varies enormously by language and by domain; models trained on news text perform poorly on transcripts and tickets |
| Cloud data-protection services (Google Cloud Sensitive Data Protection, Amazon Comprehend PII, Azure AI Language PII detection) | Broad class coverage and transformation modes without building a pipeline | A new processor in the path for the most sensitive data you hold; class definitions are theirs and change |
| A language model prompted to extract and replace | Context-dependent cases rules and NER both miss, such as an identifying anecdote with no named entity | Non-deterministic on identical input, which makes recall unmeasurable and audit statements impossible to support |
| Tokenisation vaults and format-preserving encryption | Workflows where the real value must be recoverable by an authorised reader downstream | The vault is now the crown jewel; a token that escapes into an export outlives the mapping that gave it meaning |
| Database and warehouse masking (column policies, dynamic masking, masked views) | Keeping the raw value in one governed place while everything else reads a masked projection | Protects the query path only; anything that extracted the data earlier holds the original |
| Observability redaction processors (OpenTelemetry collector processors, log pipeline filters) | Keeping traces and logs usable without them becoming a parallel copy of production data | Runs after the data has left the application, so a crash or a direct exporter bypasses it |