Guardrails and AI Safety

Guardrails and AI Safety

Content Moderation — References

Content moderation › References

28 of 57 in this book · 2 min read

A published account of operating a classifier at scale, the evidence on who gets wrongly flagged, the institutional literature on moderation as an activity, and the transparency obligations now attached to it.

Classifiers in production

  • A Holistic Approach to Undesired Content Detection in the Real World — Markov and colleagues at OpenAI, AAAI Conference on Artificial Intelligence, 2023. One of the few papers describing a deployed moderation classifier end to end, including taxonomy construction, active learning on production traffic, and the distribution shift between training data and real requests. Cited by name.
  • Llama Guard: LLM-based Input-Output Safeguard for Human-AI Conversations — Hakan Inan, Kartikeya Upasani, Jianfeng Chi and colleagues at Meta, 2023. Cited here for its separation of the taxonomy from the classifier, which is what makes a category set replaceable without retraining. Cited by name.
  • Ex Machina: Personal Attacks Seen at Scale — Ellery Wulczyn, Nithum Thain and Lucas Dixon, International World Wide Web Conference, 2017. The crowd-annotation methodology behind a widely used toxicity classifier, and a clear illustration of how annotator disagreement becomes an irreducible ceiling on measured accuracy. Cited by name.

Who the errors fall on

  • The Risk of Racial Bias in Hate Speech Detection — Maarten Sap, Dallas Card, Saadia Gabriel, Yejin Choi and Noah A. Smith, Annual Meeting of the Association for Computational Linguistics,
    1. Measured disparities in which dialects are flagged, and the demonstration that annotator instructions materially change the outcome. The strongest available argument for per-population error reporting rather than aggregate accuracy. Cited by name.

Moderation as an institution

  • Custodians of the Internet: Platforms, Content Moderation, and the Hidden Decisions That Shape Social Media — Tarleton Gillespie, Yale University Press, 2018. Why moderation is constitutive of a product rather than an add-on, and why the policy, not the classifier, is the thing being argued about. Cited by name.
  • Behind the Screen: Content Moderation in the Shadows of Social Media — Sarah T. Roberts, Yale University Press, 2019. The working conditions of commercial review at scale, which is the part of this capability that engineering plans rarely account for. Cited by name.

Obligations and commitments

  • The Santa Clara Principles on Transparency and Accountability in Content Moderation — the civil-society baseline for numbers published, notice given to affected users, and the existence of a meaningful appeal. Cited by name.
  • Regulation (EU) 2022/2065, the Digital Services Act — cited here for the statement-of-reasons and internal-complaint obligations, which turn an appeal path from a product choice into a requirement for services in scope. Cited by name; consult the official text for applicability.
  • The published usage policies of the major model providers, which are the operative category definitions for anything built on their inference and are revised without notice. Cited by name.

No verified reference is recorded for threshold derivation from category-specific error costs, for sampling the allowed stream as a miss-rate estimator, or for the residual distribution a moderation layer sees behind a model that already refuses. These are operational practices and the published work measures classifiers in isolation.