Tool Timeout and Retry

What it is

The deadline placed on a tool invocation, and the decision about what to do when that deadline fires or the call fails outright. The subject matter is dominated by one fact: tools change things. A call that times out may have already completed on the far side, and a second attempt may do the work twice. Everything here — classification of failures, idempotency keys, budget arithmetic, circuit breaking — exists to keep a recovery attempt from being worse than the failure it is recovering from.

Why it matters

Tool calls sit inside a user-visible turn, so a slow upstream becomes an unresponsive product, and the obvious remedies make things worse. A deadline too short abandons work that was about to succeed, leaving the far side to complete it unobserved. A retry policy applied uniformly duplicates payments, duplicates messages, and amplifies an upstream's bad hour into an outage. Getting this right is the difference between a system that degrades and one that causes incidents while trying to recover.

Retrying a tool is not retrying a model

A second attempt at a model call costs tokens and produces a different answer; nothing in the world changed because the first one failed. A second attempt at a tool call can send the email twice. That single difference reorders the whole design: classification has to establish whether a repeat is safe before it establishes whether the failure was transient, the unit of retry has to be something the far side can recognise as a repeat, and a tool that cannot offer that recognition is one where the correct policy is to fail and tell somebody.

Key decisions

• Where the deadline comes from. A per-call constant is easy and wrong once calls compose; a budget passed down from the turn and decremented by elapsed time is the only version that does not overrun what the user is waiting for.
• What a timeout means for the far side. Either the call is cancellable and you can say it did not happen, or it is not and the outcome is unknown — which is a different state from failure and needs its own handling.
• Which failures are retryable, decided from the tool's declared effect and idempotency rather than from the error alone. A transient network error on a non-idempotent write is still not safely retryable.
• Who generates the key that makes a repeat recognisable, and whether the far side honours it. A key the upstream ignores provides no protection whatsoever.
• Whether a retry is invisible to the model or surfaced as a result. Invisible retries hide latency; surfaced ones let the model choose a different approach instead of waiting.
• When to stop calling an upstream altogether, on what signal, and what the model is told while the breaker is open.
• What reconciles the ambiguous cases — the calls whose outcome was never established — and on what cadence, because nothing in the request path can resolve them.

Failure modes

• A payment, a message, or a provisioning action executes twice because a timeout was treated as a failure and the attempt was repeated without a key the upstream recognises.
• The client gives up, the server completes, and the two systems hold different states with nothing scheduled to notice the divergence.
• Retries are attempted on a permanently invalid request, so the same rejection is purchased three times and the eventual error arrives three times later.
• Every client retries a struggling upstream at the same moment, and the retry traffic prevents the recovery it was waiting for.
• Retry layers stack: the transport retries, the library retries, and the model retries after seeing the error, producing many more attempts than any single layer intended.
• The sum of attempts and backoff exceeds the turn budget, so the user receives a timeout from the interface while the agent is still working.
• Backoff is exponential without jitter, so a fleet that failed together retries together, in waves, indefinitely.
• A cancellation is issued on the client side only, the upstream work continues, and resources stay held until they expire on their own schedule.

Related capabilities

• Tool Rate Limiting — deliberate restraint, as opposed to reaction to failure
• Tool Registry — declares the idempotency a retry decision depends on
• Function Calling — the loop whose budget a deadline must fit inside
• Tool Result Normalization — gives a timeout a distinguishable shape in the result
• Webhook and Event Connectors — where an unresolved outcome is often reported later

Applies: Production readiness · Design questions