Abstract
Query routing, the task to route user queries to different large language model (LLM) endpoints, can be considered as a text classification problem. However, out-of-distribution queries must be handled properly, as those could be questions about unrelated domains, queries in other languages, or even contain unsafe text. Here, we thus study a \emph{guarded} query routing problem, for which we first introduce the Guarded Query Routing Benchmark (GQR-Bench), which covers three exemplary target domains (law, finance, and healthcare), and seven datasets to test robustness against out-of-distribution queries. We then use GQR-Bench to contrast the effectiveness and efficiency of LLM-based routing mechanisms (GPT-4o-mini, Llama-3.2-3B, and Llama-3.1-8B), standard LLM-based guardrail approaches (LlamaGuard and NVIDIA NeMo Guardrails), continuous bag-of-words classifiers (WideMLP, fastText), and traditional machine learning models (SVM, XGBoost). Our results show that WideMLP, enhanced with out-of-domain detection capabilities, yields the best trade-off between accuracy (88\%) and speed (<4ms). The embedding-based fastText excels at speed (<1ms) with acceptable accuracy (80\%), whereas LLMs yield the highest accuracy (91\%) but are comparatively slow (62ms for local Llama-3.1:8B and 669ms for remote GPT-4o-mini calls). Our findings challenge the automatic reliance on LLMs for (guarded) query routing and provide concrete recommendations for practical applications. GQR-Bench will be released as a Python package -- \texttt{gqr}.
Abstract (translated)
查询路由任务是指将用户查询定向到不同的大型语言模型（LLM）端点，可以视为一个文本分类问题。然而，必须妥善处理出界查询，因为这些可能是与目标领域无关的问题、其他语言的查询或包含不安全内容的查询。在此背景下，我们研究了一个受保护的查询路由问题，并为此引入了受保护的查询路由基准（GQR-Bench），该基准涵盖了三个典型的目标领域（法律、金融和医疗保健）以及七个用于测试抵御出界查询能力的数据集。 我们使用GQR-Bench对比了基于LLM的路由机制（包括GPT-4o-mini、Llama-3.2-3B和Llama-3.1-8B）、标准的LLM保护措施方法（如LlamaGuard和NVIDIA NeMo Guardrails）、连续词袋分类器（WideMLP、fastText）以及传统的机器学习模型（SVM和支持向量机、XGBoost）的有效性和效率。我们的结果表明，增强出界检测能力的WideMLP在准确率（88%）与速度(<4毫秒)之间提供了最佳的平衡。基于嵌入的fastText在速度上表现卓越（<1毫秒），尽管其准确性略低（80%）。而LLM在准确性方面表现出最高水平（91%），但相对来说运行较慢（本地Llama-3.1:8B为62毫秒，远程GPT-4o-mini调用则需669毫秒）。 我们的发现挑战了自动依赖于LLM进行受保护的查询路由的做法，并提供了实际应用中的具体建议。GQR-Bench将作为一个Python包——\texttt{gqr}——发布，供研究和开发人员使用。
URL
https://arxiv.org/abs/2505.14524
