Content Moderation — Patterns

The definition document is the source of truth

Problem. A category name is a shared illusion. Three people who agree that harassment should be blocked disagree about half the borderline items, and nothing in the system records which interpretation the threshold was set for. When the classifier changes, or the team changes, there is no artifact to compare against.

Approach. Write each category as prose with inclusions, explicit exclusions, and a set of adjudicated examples — items that were argued about and decided, with the reasoning recorded. Hold it in version control, review changes like code, and derive everything else from it: the labelling instructions, the evaluation set, the prompt if a model adjudicates, and the explanation shown to a user. A category with no adjudicated borderline examples is not ready to have a threshold.

Trade-off. Maintaining these documents is continuous work with no visible output, and the examples that make them useful are by nature the most unpleasant material in the organisation, which constrains who can work on them and where they can be stored. Precision in the definitions also exposes disagreements between legal, product, and trust teams that an ambiguous category had been comfortably hiding.

Operating points derived from cost, not from a single score

Problem. A threshold chosen to maximise a balanced metric assumes a miss and a false positive cost the same. They never do, and the ratio differs by an order of magnitude between categories — wrongly blocking a self-harm support question and wrongly allowing one are not comparable errors in either direction.

Approach. For each category, state the cost of a miss and the cost of a false positive in whatever units the organisation actually reasons in, estimate the base rate from sampled traffic, and choose the threshold that minimises expected cost rather than the one that maximises a score. Record the assumption alongside the threshold so the next person can argue with the reasoning rather than with the number. Re-derive whenever the base rate or the classifier moves.

Trade-off. The costs are estimates, often by people with an interest in the answer, and the procedure lends a spurious precision to what remains a judgement. It also produces thresholds that look indefensibly lenient for some categories when written down, which creates pressure to tighten them back towards a uniform setting that the arithmetic says is worse.

Sample what was allowed, not only what was flagged

Problem. Every signal the system generates comes from items it already suspected. The review queue measures the false-positive rate and is structurally incapable of measuring the miss rate, so a classifier degrading on a new pattern looks healthy right up until an external report arrives.

Approach. Draw a continuous random sample from the allowed stream, including items scored far below threshold, and route it to the same reviewers using the same definitions. Report the two rates together, and treat a change in the sampled miss rate as the primary quality signal, since it is the only one not conditioned on the classifier's own opinion. Size the sample from the rate you need to detect, which for rare categories is uncomfortably large.

Trade-off. Reviewing content that was allowed means deliberately exposing people to material the system saw no reason to flag, most of which is entirely ordinary and all of which costs reviewer time. For genuinely rare categories the sample size required to detect a change is often beyond what any team will fund, which limits the pattern to the categories with enough volume.

Match the classifier's window to the unit of judgement

Problem. Scoring an isolated span makes a quotation indistinguishable from an assertion and a refusal indistinguishable from compliance. Scoring an entire conversation dilutes a single severe passage into a long benign context and shifts the score below any useful threshold.

Approach. Decide, per category, what the smallest self-sufficient unit of judgement is, and feed exactly that. Some categories are properties of a span and are correctly judged on a span with a short surrounding window; others — harassment of a participant, escalating solicitation — are properties of an exchange and must be judged on the exchange. Where both apply, score both and treat them as separate signals rather than averaging them into one.

Trade-off. Multiple windows multiply the cost per item and multiply the number of thresholds to maintain. Supplying conversational context to a classifier also means sending more content to wherever it runs, which is a data-handling decision that may not be available for the vendor you chose on accuracy.