Tool Rate Limiting — Patterns

Key the limit to the scarce resource

Problem. Limits are usually written against whatever is convenient to identify — the calling service, the tool name, the endpoint. The thing that actually runs out is the upstream account's quota, which several tools and several services share, so every individual limit can be satisfied while the only one that matters is exceeded.

Approach. Model the scarce resources explicitly as named allowances — this upstream credential, this metered plan, this concurrency pool — and attach each tool to the allowances it consumes. Enforce against the allowance, with per-caller shares carved out of it, so the sum of shares is a property of the allowance rather than the result of adding up independent decisions.

Trade-off. It requires mapping tools to upstream accounts and keeping that mapping current as integrations change, which is exactly the metadata that goes stale. Shared allowances also make one team's limit change visibly affect another's capacity, which turns a configuration edit into a negotiation.

Shed early, queue only briefly

Problem. Refusing work feels worse than delaying it, so limited calls get queued. Under a sustained burst the queue absorbs everything, latency climbs past the point of usefulness, and work is eventually performed for sessions that ended long ago.

Approach. Give every queued call a deadline inherited from the turn it belongs to, drop it the moment the deadline can no longer be met rather than when it is reached, and cap queue depth at a length derived from throughput times acceptable wait. Prefer refusing at admission, with an explicit hint about when capacity is expected, to accepting work that will not be delivered in time.

Trade-off. Visible rejections are a worse experience than invisible delay right up until the queue collapses, so this pattern looks like a regression in normal conditions and only pays during the burst. It also needs a model-facing representation of "refused, try later" that does not cause the agent to abandon the task.

Two ceilings: one for the upstream, one for the run

Problem. Infrastructure limiters see aggregate traffic and have no opinion about a single agent run issuing the same call two thousand times, because the aggregate rate looks normal. Loop containment and quota protection are genuinely different constraints and one mechanism cannot see both.

Approach. Enforce the shared-resource ceiling where traffic converges, and separately give each run a budget — total calls, calls per tool, and total metered spend — enforced inside the loop and decremented as it goes. Exhausting a run budget terminates that run with a distinguishable outcome and a trace, not a retry.

Trade-off. Run budgets must be generous enough for legitimate long tasks, which makes them loose enough that a genuinely pathological loop can still do real damage before hitting one. Per-run enforcement also needs the loop to be the only path to the tools, which is not true once anything calls them directly.

Mirror the upstream's accounting, then listen to it

Problem. A limiter guessing at an upstream's remaining quota either stays so far under it that capacity is wasted, or discovers the real boundary by being rejected — spending part of the quota on failures and, with some providers, extending the penalty each time.

Approach. Configure the published limit as the starting estimate, and correct it from what the upstream actually says: read quota and reset headers from successful responses to track remaining allowance, and treat a throttling response with a wait hint as authoritative, pausing that allowance for the stated duration rather than applying a local backoff schedule. Where an upstream publishes nothing, converge on a limit by increasing gradually and backing off sharply on rejection.

Trade-off. Header conventions differ between providers and some report per-endpoint figures that cannot be combined into one allowance. A limiter that trusts upstream accounting also inherits its inaccuracies, including stale values served from the upstream's own caches.