Tool Rate Limiting

What it is

Deliberately capping how much tool traffic a caller may generate — per second, per run, per day, or in flight at once — and deciding what happens to the calls that exceed the cap. The constraint is chosen in advance and applied to healthy traffic. It exists to keep an agent inside an upstream's published quota, to stop one tenant consuming capacity the rest are sharing, to bound spend on metered calls, and to put a ceiling on how much damage a looping agent can do before anyone notices.

Why it matters

An agent generates load in a shape nothing else does: a single user request can expand into dozens of calls, and a planning error can expand it into thousands with no human pacing the sequence. The upstreams being called are usually third parties with quotas that, once exhausted, are exhausted for every user of that account — so one runaway session can remove a product feature for everybody. Limits are also the only control that bounds cost on metered tools, where the bill arrives long after the traffic.

Limiting on purpose versus reacting to failure

A limit is applied to calls that would otherwise have succeeded, by a system that has decided in advance how much is reasonable. Reacting to a call that timed out or came back rejected is a different mechanism with different arithmetic, and the two interact badly when confused: a retry policy that treats a self-imposed limit as a transient failure will simply retry into it, and a limiter that counts rejected attempts against the quota will throttle a caller for the upstream's bad behaviour. Keep them separate, and make a limit decision distinguishable in the result from an upstream failure.

Key decisions

• What the counter is keyed on. The useful key is whatever is genuinely scarce — usually the upstream account or credential, not the tool name and not the calling service.
• Which dimensions are capped. Requests per interval protects a published quota; concurrency protects an upstream from saturation; calls per agent run bounds a loop; spend per tenant bounds the bill. Most deployments need several, and they fail differently.
• Behaviour at the limit: shed immediately, queue with a bounded wait, or degrade to a cheaper path. Queueing converts a rejection into unbounded latency and is the default choice that most often goes wrong.
• Whether the upstream's own published limits are mirrored locally or discovered by being rejected. Mirroring needs maintenance and keeps you inside the quota; discovery is free and means routinely spending the quota on rejections.
• Whether enforcement is per process or shared. Per-process limits multiply by replica count, which silently becomes the real limit.
• How a limited call is reported to the model, and whether it is told when capacity returns.
• How fairness is expressed when several tenants share one upstream credential, which is the situation where limits matter most and are hardest to get right.

Failure modes

• The limit is enforced in each process, the service scales to twelve replicas, and the effective ceiling is twelve times what anyone intended.
• An agent receives a rejection and immediately calls again, consuming the recovery window it was being asked to wait through.
• Limited calls are queued rather than refused, the queue grows during a burst, and requests are served long after the user who made them has gone.
• A batch job and interactive sessions share one upstream credential, and the batch exhausts the quota every morning.
• The limit protects the upstream from overload while doing nothing about cost, and the first signal of a problem is an invoice.
• A server-supplied wait hint is ignored in favour of a local schedule, and the caller stays throttled far longer than necessary.
• Caps are defined per tool while the actual constraint is a shared account, so ten tools each within their limit together exceed the one that matters.
• A limit rejection is reported to the model as a generic tool error, and the model concludes the action is impossible and abandons a task that would have succeeded in thirty seconds.
• Nothing caps calls per agent run, so a planning loop issues the same call ten thousand times within every individual limit.

Related capabilities

• Tool Timeout and Retry — the reactive counterpart this must not be confused with
• MCP Gateway — the natural enforcement point when many callers share upstreams
• Tool Permission Management — decides whether a call is allowed at all
• Tool Registry — records which upstream account a tool actually consumes
• Function Calling — the loop whose expansion makes these limits necessary

Applies: Production readiness · Design questions