Tool Discovery — Patterns

Index intent, not the author's description

Problem. Retrieval over descriptions compares a user's phrasing against a tool author's phrasing, and those two populations do not share vocabulary. A perfectly good tool is missed because it is described in the language of the system it wraps.

Approach. Build the retrieval index from a generated intent document per entry rather than from the raw description: several paraphrased statements of when the tool is the right choice, the question forms it answers, and the arguments a typical call carries. Generate it once at admission, review it as part of onboarding, and store it as part of the entry so retrieval behaviour is reproducible and auditable.

Trade-off. Generated text drifts from the implementation it describes and nobody notices, because the only symptom is a gradual decline in recall. The documents also need regenerating whenever the entry changes, which puts a model call into an administrative path that was previously free.

Two-stage narrowing through a source boundary

Problem. Flat retrieval across thousands of entries from dozens of backing systems returns candidate sets containing the same operation from five different systems, and the model picks whichever appeared first.

Approach. Select the backing system or domain first, from a short list of descriptions written about systems rather than operations, and retrieve within it only after that choice. The first stage is cheap, highly accurate when the systems are genuinely distinct, and reduces the second stage to a problem small enough to solve well.

Trade-off. A wrong first-stage choice cannot be recovered from in the second, and models rarely volunteer that none of the offered operations fit. The split also fails for tasks that genuinely span systems, which need an explicit escape to flat retrieval.

Enumerate in the background, serve from a snapshot

Problem. Contacting every source when a session starts makes the first response as slow as the slowest source and as reliable as the least reliable one, and the user is waiting throughout.

Approach. Run enumeration as a background process on its own schedule, writing a versioned snapshot of the combined surface. Sessions read the snapshot and start immediately. Change notifications from sources trigger an early refresh rather than an inline fetch, and a source that fails enumeration keeps its previous entries with a recorded staleness marker instead of vanishing.

Trade-off. Everything served is by definition slightly out of date, so an entry removed upstream remains selectable until the next refresh — which means call-time failures must be handled gracefully and fed back as an invalidation signal. Sessions that genuinely need per-user surfaces cannot use a shared snapshot at all.

Measure the miss, not just the hit

Problem. Candidate-set quality is usually assessed by whether the chosen tool worked, which only ever examines tools that were offered. The expensive failure — the right tool was never in the set — leaves no trace, because the model proceeds with something else.

Approach. Maintain a labelled set of requests with their known-correct entry and score recall at the candidate-set size actually used in production, as a gate on any change to the index, the embedding, or the thresholds. Separately, sample sessions where the model declined to call anything or called something that returned an irrelevant result, and check whether a suitable entry existed but was not offered.

Trade-off. The labelled set is manual work that decays as the surface changes, and it systematically under-represents the newest entries, which are the ones most likely to be missed. It is still the only way to see a failure that otherwise leaves no signal at all.