Abstract
Existing benchmarks that assess Language Models (LMs) as Language Agents (LAs) for tool use primarily focus on stateless, single-turn interactions or partial evaluations, such as tool selection in a single turn, overlooking the inherent stateful nature of interactions in multi-turn applications. To fulfill this gap, we propose \texttt{DialogTool}, a multi-turn dialogue dataset with stateful tool interactions considering the whole life cycle of tool use, across six key tasks in three stages: 1) \textit{tool creation}; 2) \textit{tool utilization}: tool awareness, tool selection, tool execution; and 3) \textit{role-consistent response}: response generation and role play. Furthermore, we build \texttt{VirtualMobile} -- an embodied virtual mobile evaluation environment to simulate API calls and assess the robustness of the created APIs\footnote{We will use tools and APIs alternatively, there are no significant differences between them in this paper.}. Taking advantage of these artifacts, we conduct comprehensive evaluation on 13 distinct open- and closed-source LLMs and provide detailed analysis at each stage, revealing that the existing state-of-the-art LLMs still cannot perform well to use tools over long horizons.
Abstract (translated)
现有的评估语言模型（LM）作为语言代理（LA）在工具使用方面的能力的基准测试主要集中在无状态、单一回合交互或部分评估上，例如单次回合内的工具选择，忽视了多轮应用中互动固有的有状态性质。为了填补这一空白，我们提出\texttt{DialogTool}，这是一个多轮对话数据集，包含了工具使用的整个生命周期中的有状态工具互动，涵盖了六个关键任务，在三个阶段内：1）\textit{工具创建}; 2) \textit{工具利用}: 工具意识、工具选择和工具执行；3) \textit{角色一致的响应}: 响应生成和角色扮演。此外，我们构建了\texttt{VirtualMobile}——一个模拟API调用并评估所创造API的鲁棒性的身临其境虚拟移动评估环境（在本文中我们将交替使用工具和API这两个术语，并且它们在此文中没有显著差异）。利用这些资源，我们在13种不同的开源和闭源大规模语言模型上进行了全面评估，并提供了每个阶段的详细分析，揭示了现有的最先进的LLM仍然无法很好地处理长时间跨度内的工具使用。
URL
https://arxiv.org/abs/2505.13328
