Tool Selection Evaluation

What it is

Measuring whether the system reached for the right capability, with the right arguments, at the moment it decided to act. Three error classes live here and need separating: choosing the wrong tool from the available set, inventing one that does not exist, and choosing correctly while populating the arguments wrongly. The awkward property of all three is that the evidence is counterfactual — knowing the choice was wrong requires knowing what a different choice would have produced, and the only record available is of the branch that was actually taken.

Why it matters

A wrong selection is the cheapest failure to cause and among the most expensive to detect. The chosen tool usually returns something, the model summarises it fluently, and the answer is wrong in a way that reads fine; nothing in the response indicates that a different capability held the real answer. Selection quality is also the property most sensitive to changes nobody treats as risky — adding a tool, renaming one, rewording a description — and it degrades as the catalogue grows, which is the direction every product moves. Separating selection from execution is what lets a team tell "the agent called the wrong thing" apart from "the thing it called is broken", and those are different backlogs owned by different people.

Key decisions

• Whether a single correct tool is assumed per case, or a set of acceptable ones. In real catalogues several tools genuinely overlap, and a strict label penalises a defensible choice.
• How the no-call case is treated. Answering from existing knowledge is often correct, and a suite that scores it as a missed call teaches the system to call something every time.
• Whether argument correctness is scored separately from tool identity, and at what granularity — the whole argument object, or per field, since one wrong field is a different defect from a wrong tool.
• What catalogue the evaluation presents. Selection accuracy measured against six tools says nothing about the same model with sixty, so the catalogue is part of the measurement.
• Whether the evaluation stops at the decision or executes the call. Stopping is cheap, reproducible and blind to arguments that are well-formed and semantically wrong; executing requires a sandbox and makes the result depend on the tool's behaviour.
• How invented tools and invented parameters are counted, since they are a distinct and particularly diagnostic failure rather than just another wrong answer.
• How often the labelled set is reconciled against the live catalogue, given that a renamed or withdrawn tool silently turns every item referencing it into a false failure.

Failure modes

• Two tools overlap in purpose, the label names one, and the model is marked wrong half the time for a choice the product considers fine.
• A tool is renamed in the catalogue and not in the evaluation set, so a suite that was passing reports a total collapse in selection accuracy overnight.
• Selection and execution are scored together, so a correct choice to a tool that is returning errors is recorded as a selection defect and the investigation starts in the wrong place.
• The suite rewards calling a tool, the system learns to call one for every request, and latency and cost rise while answers get no better.
• Accuracy is measured with ten tools in the prompt and the product ships with ninety, where the descriptions no longer fit the window and selection degrades in ways the suite cannot see.
• Arguments are compared as whole objects, so an otherwise perfect call with a date in the wrong format is indistinguishable from a call to the wrong tool.
• Hallucinated tool names are counted as ordinary misses, hiding a signal that the catalogue presentation itself is the problem.
• Evaluation items were generated from the tool descriptions, so each case is answerable by matching wording and the suite measures paraphrase recognition rather than selection.
• The system picks a correct tool and then ignores the result, which a selection-only measurement scores as a complete success.

Related capabilities

• Agent Trajectory Evaluation — where repeated selections are judged as a sequence rather than individually
• Decision Model Evaluation — the per-class and threshold arithmetic that applies once selection is framed as a choice over a finite set
• Deterministic Evaluation — validates argument structure before any semantic judgement is needed
• Regression Evaluation — what catches a catalogue change that quietly moved the decision boundary
• Golden Datasets — holds the labelled cases and the acceptable-tool sets they depend on

Applies: Production readiness · Design questions