Model Capability Registry — Patterns

Declared and observed fields kept apart

Problem. A single record mixing "the provider says 200k context" with "we measured p95 at 4.2 seconds last Tuesday" has no coherent refresh policy, and a consumer cannot tell which values it should trust during an incident.

Approach. Split each entry into a declared section sourced from documentation and an observed section sourced from your own traffic and probes, each with its own timestamp and source attribution. Consumers choose explicitly: a routing predicate reads declared support, a latency-sensitive decision reads the observed percentile.

Trade-off. Two sections mean two refresh mechanisms and the possibility of visible disagreement between them. That disagreement is diagnostic information — a declared capability that probes fail is exactly what you want surfaced — but someone must act on it, or the registry just displays its own contradictions.

Capability predicates instead of capability booleans

Problem. supports_tools: true is satisfied by models that differ on parallel calls, schema depth, required-field enforcement, and whether a tool result may be an image. Code that branches on the boolean breaks on the differences.

Approach. Model capabilities as a set of narrow, independently testable assertions — parallel tool calls, strict schema adherence, nested object parameters, image inputs in tool results, streaming tool-call deltas — each verifiable by a probe. Consumers express requirements as the conjunction they actually need, not as a coarse flag.

Trade-off. The assertion set grows as providers add features, and every new assertion is unknown for every existing entry until someone probes or researches it. Unknown must be a distinct value from false, and consumers must decide what to do with it.

Scheduled re-confirmation with an expiry

Problem. An entry is accurate when written and silently decays. Nothing in the system distinguishes a fact confirmed yesterday from one copied from a launch blog two years ago.

Approach. Give every field a last-confirmed timestamp and a maximum age appropriate to its volatility — prices and retirement dates short, tokenizer family effectively indefinite. A scheduled job re-runs probes for observed fields, re-fetches what provider APIs expose, and raises a task for the rest. Entries whose critical fields are past their maximum age are reported as untrusted.

Trade-off. It generates recurring human work that will be deprioritised, so the expiry policy must have teeth that are proportionate: refusing to route to an entry with a stale price is defensible, refusing because a latency measurement aged out is not.

Registry snapshots shipped with the deployment

Problem. Reading a central registry on the request path adds a dependency whose outage stops all inference, for data that changes roughly monthly.

Approach. Resolve the registry at build or deploy time into an immutable snapshot that ships with the service, with the snapshot's version recorded in telemetry. Runtime lookups hit the snapshot. A separate, non-blocking refresh path can warn that the snapshot is behind, and an urgent correction — a withdrawn model, a wrong price — is shipped as a normal expedited deploy.

Trade-off. Corrections propagate at deployment speed, which is too slow for a genuine emergency; keep a narrow runtime override channel for withdrawals specifically, and accept that it is a second source of truth with all the hazards that implies.