Model Management and Inference

Model Management and Inference

Model Routing — References

Model routing › References

12 of 49 in this book · 1 min read

Sources on cost-quality routing, cascades, and the measurement a routing threshold rests on.

Research

  • FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performance (Chen, Zaharia, Zou, 2023), arXiv:2305.05176 — LLM cascades and the cost-quality trade-off stated as an explicit optimisation.
  • RouteLLM: Learning to Route LLMs with Preference Data (Ong et al., 2024), arXiv:2406.18665 — training routers from preference data, and the cost-versus-quality framing used to report router performance.

Implementations

  • semantic-router — embedding-similarity routing against labelled exemplar sets, useful as a concrete reference for threshold tuning.

Measurement

  • Stanford CRFM HELM — multi-metric model evaluation, including cost and efficiency alongside accuracy, which is the shape a routing decision needs.

Routing decisions should be recorded as span attributes rather than only in application logs; the generative-AI section of the OpenTelemetry semantic conventions defines where the model identity belongs on a trace.