Evaluation and Quality Engineering
Tool Selection Evaluation — Stack
Tool selection evaluation › Stack
Technology landscape as of 2026-10-10. Knowledge cutoff May 2026.
Selection criteria
The components either capture the decision, score it against a label, or provide a safe place to execute the chosen call. What distinguishes them is how much of the real catalogue they can present and how precisely they localise a defect.
- Catalogue fidelity. Whether the evaluation can present the production tool surface, in the same serialisation and the same order, since all three affect the choice.
- Argument-level comparison. Whether the scorer can report which field was wrong rather than returning one verdict for the call.
- Acceptable-set labels. Whether an item can declare several correct tools, or forces one.
- No-call representation. Whether declining to call is expressible as a labelled correct outcome.
- Execution isolation. For suites that run the chosen call, whether it is sandboxed, resource-limited and reset between items.
- Catalogue reconciliation. Whether the tooling can detect that a labelled tool no longer exists, before the suite reports it as a quality regression.
- Scale behaviour. Whether the harness can vary catalogue size deliberately, so the degradation curve as tools are added is measurable rather than discovered in production.
Options
| Option | Best for | Watch out for |
|---|---|---|
| Schema validators over the emitted call (JSON Schema, the tool definitions themselves, Pydantic-style models) | Catching malformed and type-invalid arguments for free, before any judgement | Structural validity says nothing about whether the values are right |
| Published function-calling benchmarks and their harnesses (the Berkeley function-calling leaderboard and comparable suites) | Methodology reference and model shortlisting on standardised catalogues | Their catalogues are not yours; a strong published number does not transfer to a sixty-tool internal surface |
| Evaluation-framework tool metrics (DeepEval's tool-correctness scorers, agent evaluation modules in comparable libraries) | Getting per-item tool and argument comparison running without writing the comparison logic | Definitions of a correct call vary; most assume a single correct tool per item |
| Mock and record-replay tool servers | Deterministic, free execution of the chosen call, with controllable failure injection | Mocks drift from the real tools; a suite passing against mocks proves nothing about the live surface |
| Sandboxed execution runners (containerised runtimes, ephemeral service fixtures) | Scoring by observable effect, which catches semantically wrong arguments that validate fine | Infrastructure cost, isolation requirements, and a reset path that has to be genuinely reliable |
| Trace-based analysis over production tool calls (LLM observability platforms and OpenTelemetry-backed stores) | Measuring real selection behaviour, including on the catalogue as it actually ships | No labels; defects must be inferred from downstream signals such as immediate re-calls and error rates |
| Catalogue-aware retrieval layers evaluated separately (tool retrieval and shortlisting components) | Products where the full catalogue cannot fit in the prompt and a retrieval step precedes selection | Introduces a second recall problem upstream; a selection failure may be a retrieval failure wearing a disguise |