Content Moderation

What it is

Judging text, images, and audio against a defined set of harm categories, on both sides of the conversation, and acting on the judgement. The substance of the capability is the taxonomy and its operating points: what each category means in writing, what severity within it looks like, what a reviewer needs in order to agree, and where the threshold sits for each category independently.

Why it matters

The categories are not a technical artifact. They are what the organisation has told its users, its regulators, and its enterprise customers it will not produce or permit, and a classifier is only the mechanism for keeping that promise. Without written definitions and adjudicated examples a category is a word, and two engineers setting a threshold for it are not setting the same threshold. The stakes are asymmetric in a way that resists a single operating point: the rare miss is the one that appears in a news article, and the common false positive is the one that disproportionately silences the people least able to complain about it.

Key decisions

• Whether the taxonomy is adopted from an external scheme or written for the product. An adopted one arrives with labelled data and does not match your policy; a bespoke one matches and has no data behind it.
• Thresholds and actions per category rather than globally. Collapsing categories whose costs differ by orders of magnitude into one score discards the only structure that matters.
• Whether what the user sends and what the system produces are scored by the same model at the same settings. The organisation authors one of them and merely transmits the other.
• How much context the classifier receives: the span, the surrounding turn, or the conversation. A quotation, a report, a refusal, and an endorsement can be the same string.
• What happens to a flagged item — blocked, delivered behind an interstitial, downranked, routed to review — and what the person on the other end is told.
• Whether an appeal exists, who answers it, how long it takes, and whether a successful appeal changes anything about the classifier.
• Coverage per language and per modality, reported separately, and what the product does in a language the classifier was never evaluated on.
• Who looks at flagged material, in what volume, and what protects them.

Failure modes

• Thresholds are set on a benchmark where a tenth of the items violate and deployed against traffic where one in ten thousand does, so the review queue fills with items that look correct until read.
• Quoted, reported, and counter-speech uses of the same words score as the thing they describe, which removes material from the communities discussing harassment directed at them.
• A clinical, legal, or harm-reduction product is scored by a general classifier whose categories overlap its purpose, so the product's core function reads as a stream of violations.
• Performance is reported as one averaged accuracy figure, concealing that behaviour in half the supported locales is close to chance.
• The classifier is upgraded, the thresholds are not re-derived against the new score distribution, and the block rate moves by a factor while the policy has not changed at all.
• Images and audio are described by a model and the description is scored, so what is judged is a caption rather than the content, and anything the captioner declines to describe passes.
• Flagged items are retained indefinitely to improve the classifier, which builds exactly the archive the policy exists to prevent.
• Reviewers see raw flagged content with no rotation, no volume limits, and no support, and the cost of the control is paid entirely by the people operating it.
• The same model scores the content and writes the explanation shown to the user, so a wrong decision arrives with a fluent and entirely fabricated justification.

Related capabilities

• Policy Enforcement — the organisation's own rules, which are not harm categories
• Jailbreak Detection — the attempt to elicit, as distinct from what was produced
• Output Guardrails — the checkpoint where a judgement about generated content is applied
• Human Approval Gates — the review machinery an uncertain band depends on
• Fail-closed Behavior — what happens to content when the classifier is unavailable

Applies: Production readiness · Design questions