Tools MCP and External Integration
Tool Rate Limiting — Stack
Tool rate limiting › Stack
Technology landscape as of 2026-10-10. Knowledge cutoff May 2026.
Selection criteria
The decision is mostly about where counting happens and what it can count, not about which algorithm is implemented.
- Shared state. Whether counters are global across replicas or local to a process. Local counting is simple, fast, and gives an effective limit that changes with autoscaling.
- Key flexibility. Whether a limit can be keyed on a composite — tenant and upstream credential and tool — and whether new keys can be introduced without a deployment.
- Dimension coverage. Whether requests per interval, concurrent in-flight calls, and a cost or token-weighted budget can all be expressed, since these constrain different scarcity.
- Decision cost. Added latency per call, which for a centrally coordinated limiter includes a network round trip on every single invocation.
- Behaviour under partition. What the limiter does when its shared store is unreachable: fail open and risk exhausting an upstream quota, or fail closed and stop all tool use.
- Hint propagation. Whether an upstream's own throttling response and wait hint can be fed back into local state, so the limiter learns the real remaining quota rather than guessing.
- Fairness model. Whether the mechanism can prevent one caller from monopolising a shared allowance, or only enforce per-caller ceilings that collectively exceed the shared one.
- Introspection. Whether a caller can ask how much allowance remains before spending it, which is what makes graceful degradation possible rather than reactive.
Options
| Option | Best for | Watch out for |
|---|---|---|
| In-process token bucket | Single-replica services and per-run loop caps where exactness does not matter | Multiplies by replica count; resets on deploy, which turns a rollout into a burst |
| Centralised counters in a shared store (Redis-backed sliding windows, dedicated limiter services) | Fleet-wide enforcement against a shared upstream quota | A round trip per call, and a hard dependency whose failure mode must be chosen deliberately |
| Gateway or proxy rate-limit plugins (Envoy rate limit service, Kong, NGINX and similar) | Applying one policy across many callers without touching application code | Keyed on transport attributes; expressing "this tenant on this upstream credential" often requires header plumbing |
| Concurrency limiters and bulkheads rather than rate counters | Protecting an upstream from saturation, where simultaneous calls matter more than their rate | Does not bound total volume or cost; a slow upstream silently reduces throughput to nothing |
| Queue-based admission with bounded depth and shedding | Smoothing bursts where some delay is acceptable and work is deferrable | Unbounded or over-generous queues convert rejection into latency; depth must be small and enforced |
| Adaptive limiting from observed upstream responses (AIMD-style controllers) | Upstreams whose real limits are undocumented or vary | Learns by being rejected, so it spends part of the quota discovering it; needs careful damping |
| Per-run call budgets enforced in the agent loop | Containing planning loops, which no infrastructure limiter sees as abnormal | Only bounds one run; a hundred concurrent runs each within budget still overwhelm an upstream |