Guardrails and AI Safety
Action Risk Classification — Stack
Action risk classification › Stack
Technology landscape as of 2026-10-10. Knowledge cutoff May 2026.
Selection criteria
This is more a schema and a calibration practice than a product category, so the criteria are about where the inputs come from and whether the scale can be changed without breaking its consumers.
- Input provenance. Whether the mechanism can read argument values and query the target system, or only consult static metadata registered at build time.
- Dimension independence. Whether the dimensions stay separable in the output, so a consumer can threshold on irreversibility alone rather than on a blended number.
- Scale stability. Whether the scale can be recalibrated without silently moving every downstream threshold, which usually means versioning the scale itself.
- Evaluation cost. Added latency per call; a valuation that queries the target is far more expensive than one that reads a table.
- Auditability. Whether the inputs, the version of the scale, and the resulting value are recorded together.
- Authoring. Who assigns levels, and whether the mechanism supports review of an assignment the way code review works.
- Back-testing. Whether past actions can be re-valued under a revised scale, which is what makes calibration against incidents possible at all.
Options
| Option | Best for | Watch out for |
|---|---|---|
| Static metadata on the tool registration (declared effect, reversibility flag, external-visibility flag) | A first implementation that costs nothing and separates obviously destructive tools from read-only ones | A constant per tool cannot distinguish the calls within it, which is where almost all the variance lives |
| Rule tables over argument predicates | Expressing the real distinctions — amount bands, recipient class, record counts — in a form a reviewer can read | Grows exception by exception until the table encodes history rather than policy |
| Valuation computed from the target system (matched row counts, quoted totals, recipient-list sizes) | Actions where reach is genuinely unknowable from the arguments alone | A query against the target on the critical path, and a value that is stale the instant it is computed |
| Policy engine expressions reusing the enforcement rule language | Keeping valuation and enforcement in one reviewable, versioned artifact | Strong pull towards letting the same rule both value and decide, which is the coupling worth avoiding |
| Workflow engine step annotations | Multi-step plans where accumulation across steps is the thing being measured | Only covers actions the engine orchestrates; anything the agent does directly is unvalued |
| Learned scoring from incident and reversal history | Organisations with enough recorded outcomes for the scale to be evidence rather than opinion | Needs labelled incidents most teams do not have, and an opaque value is hard to defend in a review |
| Adopted external risk taxonomies (NIST AI Risk Management Framework categories, Open FAIR factors) | Aligning the scale with the vocabulary an auditor or insurer already uses | Written for organisational risk over a year, not for a single tool call in two hundred milliseconds |