Evaluation and Quality Engineering

Evaluation and Quality Engineering

Tool Selection Evaluation — References

Tool selection evaluation › References

40 of 73 in this book · 2 min read

Material on choosing a callable from a catalogue, on what happens when the catalogue outgrows the prompt, and on where invented calls come from.

Choosing from a large surface

  • Gorilla: Large Language Model Connected with Massive APIs (Patil et al., 2023), arXiv:2305.15334 — invocation accuracy against a large API surface, and hallucinated API arguments treated as a measurable error class in their own right rather than as ordinary mistakes.
  • ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIs (Qin et al., 2023), arXiv:2307.16789 — what happens to selection when the candidate set is far larger than a prompt can hold, and the retrieval stage that becomes necessary as a result.

Learning when to call at all

  • Toolformer: Language Models Can Teach Themselves to Use Tools (Schick et al., 2023), arXiv:2302.04761 — how a model learns when a call is worthwhile, which is the framing behind treating the no-call outcome as a labelled correct answer rather than as a miss.

Standardised measurement

  • The Berkeley function-calling leaderboard and its evaluation harness are the most widely used public comparison of function-calling behaviour, including categories for parallel calls and for cases where no call should be made. Described by name without a specific paper title, since the project is maintained as an evolving leaderboard rather than a single publication.

The arithmetic of a rare wrong choice

  • The Base-Rate Fallacy and its Implications for the Difficulty of Intrusion Detection — Stefan Axelsson, ACM Conference on Computer and Communications Security, 1999. Applicable here because selection errors are rare relative to total calls, which determines how large a sample must be before an accuracy difference between two catalogues means anything. Cited by name.

No verified reference is recorded for acceptable-set labelling with preference ordering, for degradation curves against catalogue size, or for reconciling evaluation labels against a live tool registry.