Guardrails and AI Safety
Content Moderation — Stack
Content moderation › Stack
Technology landscape as of 2026-10-10. Knowledge cutoff May 2026.
Selection criteria
The decisive properties are about the taxonomy and the evidence behind it, not about headline accuracy on someone else's test set.
- Taxonomy fit and mutability. How closely the supplied categories map to your written policy, and whether you can add, split, or disable one.
- Per-category scores. Whether the component returns a score per category or a single verdict, which determines whether you can set different operating points at all.
- Per-language evidence. Published or obtainable performance for each language you serve, separately, with the evaluation set described.
- Modality coverage. Native scoring of images and audio against direct evaluation of a generated description, which are different systems with different blind spots.
- Context intake. Whether a conversation or a surrounding turn can be supplied, which is what separates a quotation from an assertion.
- Review integration. Whether flagged items can be exported with their scores, their spans, and their model revision into a queue a human can work.
- Data handling. Where flagged content is retained, for how long, and whether it feeds the supplier's training.
- Change notification. Whether model and policy updates are announced before they land, since an unannounced update silently moves every threshold you set.
Options
| Option | Best for | Watch out for |
|---|---|---|
| Hosted moderation endpoints from model providers | A credible, maintained baseline across many categories at very low cost per call | A fixed taxonomy aligned to the provider's policy, not yours; thresholds shift when the model is updated |
| Cloud content-safety services (Azure AI Content Safety, Google Cloud safety filters, Amazon Bedrock Guardrails) | Products needing severity levels, multimodal input, and contractual terms about data handling | Severity scales are defined by the vendor and are not comparable between vendors |
| Self-hosted guard models (Llama Guard, ShieldGemma, Granite Guardian, WildGuard) | Keeping content inside your boundary and applying one judgement across several serving models | Capacity, upgrades, and evaluation become yours; published category sets still may not match your policy |
| Specialised toxicity and abuse classifiers (Perspective API, Detoxify and similar) | Narrow, well-studied dimensions such as insult and threat, with a long measurement history | Built for public comment streams; the distribution is nothing like assistant traffic, and the known demographic biases are well documented |
| A general model prompted with your own written policy | Bespoke categories where no off-the-shelf taxonomy fits, and where the rationale must be inspectable | Costly per item, non-deterministic across runs, and the rationale it writes is persuasive whether or not the verdict is right |
| Rules and lexicons maintained in-house | Named entities, campaign-specific strings, and incident response within minutes | Ages badly, generalises not at all, and is the component most likely to produce embarrassing false positives |
| Human review platforms and queueing tools | The uncertain band, appeals, and the labelled data every other option depends on | Reviewer welfare, rotation, and retention are the real design work, and they are rarely budgeted |
Models
The classifier you deploy will see a distribution no public benchmark resembles. A serving model has already refused the easiest cases, so what reaches the moderation layer is the residual: the ambiguous, the contextual, and the deliberately constructed. A model selected on benchmark accuracy is therefore selected on performance over the population it will rarely see. The practical procedure is to label a few thousand items sampled from your own traffic — including a deliberate oversample of the uncertain band — and choose both the model and the per-category threshold from measured cost on that set, where cost means your actual cost of a miss against your actual cost of a false positive, which differ by category and are never equal. Prefer a classifier from a different family than the serving model, because a guard that shares the generator's training shares its view of what is acceptable, and the cases where they agree wrongly are exactly the ones nothing else will catch. Per-language performance must be measured separately and reported separately; a single figure is an average weighted by whichever language dominated the evaluation set. For images and audio, prefer native scoring over describing the content and scoring the description, since the captioner's own refusals become silent passes. Finally, pin the revision and keep the labelled set, because the thresholds are properties of a score distribution that moves with every model update.