Tool Selection Evaluation — Patterns

Label an acceptable set, with a preference order

Problem. Mature catalogues contain tools that genuinely overlap — a general search and a specialised lookup that both answer the question, one faster and one more precise. A single correct label makes the suite report failures the team would not have raised in review, and the usual response is to delete the ambiguous items, which removes the hardest and most realistic cases.

Approach. Allow each item to carry a set of acceptable tools with an explicit ordering, and score in two parts: whether the chosen tool is in the set at all, and whether it was the preferred member. The first is the correctness metric and gates; the second is an optimality metric and informs. Record the reason an alternative is acceptable on the item, so the label can be revisited when the catalogue changes.

Trade-off. Acceptable sets are far more work to author and to keep current, and the preference order encodes a judgement that will date. A permissive set also hides genuine degradation: a model drifting from the preferred tool to the merely acceptable one shows up only in the secondary metric nobody gates on.

Score the call in three independent parts

Problem. One verdict per call conflates failures with completely different causes. A wrong tool means the descriptions or the routing logic are at fault; a hallucinated tool means the catalogue presentation is; a wrong argument means the schema or the extraction is. Collapsing them produces a single number that no single team can act on.

Approach. Emit three fields per item: tool identity against the acceptable set, existence of the named tool in the catalogue, and argument correctness evaluated per field with types, required-field presence and value correctness separated. Report them as three series. A regression then arrives pre-triaged, and the per-field argument breakdown usually points at one parameter that one recent description change made ambiguous.

Trade-off. Three metrics invite being averaged into one by whoever builds the summary, and per-field argument comparison needs a per-field notion of equality — dates, identifiers, free text and enumerations all compare differently, and the comparison rules become their own small maintenance burden.

Measure the degradation curve against catalogue size

Problem. Selection accuracy is reported against whatever catalogue the suite was built with. The product then grows from twelve tools to seventy, the descriptions stop fitting comfortably, near-duplicates appear, and accuracy falls — discovered in production, because the suite's catalogue never grew.

Approach. Run the same labelled items at several catalogue sizes: the true set, the true set plus plausible distractors, and the full production surface. Report accuracy at each. The shape of the curve is the actionable artefact — a flat curve means the catalogue is well separated, a cliff at a particular size means descriptions have started colliding and names them. Add every new tool to the distractor pool as it ships, whether or not it has items of its own.

Trade-off. It multiplies suite runtime by the number of catalogue configurations, and constructing a realistic distractor pool is ongoing work. The curve also measures the catalogue's presentation as much as the model's ability, which makes it hard to attribute a change when both moved in the same release.

Reconcile labels against the live catalogue before every run

Problem. Tools are renamed, deprecated, merged and re-scoped by the teams that own them, without reference to an evaluation set somewhere else. The suite keeps running, every item referencing a changed tool fails, and the dashboard reports a quality collapse that is entirely an identifier mismatch.

Approach. Make the first stage of the run a reconciliation: resolve every labelled tool name and every labelled argument field against the catalogue as it currently exists, and fail the run loudly with a list of unresolvable references rather than scoring the items. A rename becomes a one-line fix found in seconds instead of a day of investigation, and the reconciliation report doubles as a change log of the tool surface.

Trade-off. It blocks the suite on catalogue drift, which is correct and is also an interruption at the worst time. Reconciliation can only check that names resolve, not that a tool's semantics are unchanged — a tool keeping its name while its behaviour is redefined passes reconciliation and invalidates the labels anyway.