Function Calling — Patterns

Narrow the declaration set before the turn

Problem. Every declaration occupies prompt space and adds a distractor. Past a few dozen tools, selection accuracy falls faster than the added coverage helps, and the decline is invisible because wrong-tool calls still return plausible results.

Approach. Decide the visible set per turn rather than per session. Start from the task statement and the conversation so far, select a bounded candidate set, and expose only that. Where a group of tools belongs to one system, expose a single entry point first and reveal its members only after the model has committed to that system.

Trade-off. Narrowing introduces its own error: a tool that was never shown cannot be chosen, and the resulting failure is silent — the model answers without it rather than reporting that it was unavailable. The selection step also costs latency, and it needs its own recall measurement, which is easy to skip.

Bounded repair with a changing prompt

Problem. When a call is rejected, handing the error back and asking again often produces the identical call. The loop then burns tokens until a turn limit fires, and the user waits for a failure that was determined on the first attempt.

Approach. Cap the attempts at a small number, and make each attempt differ from the last by something other than the error text: restate the offending field's constraint in the declaration, drop the optional fields that were malformed, or narrow the call mode to the specific tool with a reduced schema. Detect a repeated identical invocation and treat it as terminal immediately rather than waiting for the cap.

Trade-off. Reconstructing the declaration mid-turn invalidates any prompt cache covering the tool block, so the cheaper repair path is also the one that loses the discount. A hard cap also converts some recoverable cases into user-visible failures.

Results re-enter context under an explicit budget

Problem. Tool results accumulate. A conversation with eight calls carries eight results into every subsequent turn, and the largest of them usually contributes nothing after the turn that requested it.

Approach. Give the loop a budget for result tokens and spend it deliberately: keep the most recent result in full, replace older ones with a short retained projection plus a handle that can fetch the full payload again if needed, and record which fields were actually referenced in the following assistant turn so the projection is informed by use rather than guessed.

Trade-off. Rewriting earlier turns breaks prefix caching and can confuse models that notice the transcript changing beneath them. Keeping a fetch handle means the full result must be stored somewhere with a lifetime at least as long as the conversation.

Executing calls the model did not choose

Problem. Some calls are not genuinely discretionary. Looking up the current account state before answering a question about it is required every time, but a model asked to decide will sometimes skip it and answer from the conversation.

Approach. Lift those out of the model's choice entirely. Execute them deterministically before the turn begins, inject the results as context, and leave only genuinely optional tools in the declaration set. The model is then choosing among things where its judgement adds value rather than being trusted to remember a policy.

Trade-off. Pre-fetching pays for calls that some turns did not need, and it shifts latency onto every request including the ones that would have answered immediately. It also moves a piece of the agent's behaviour into application code, where it is tested properly but no longer adapts.