Abstract
LLM-based function calling enables intelligent agents to interact with external tools and environments, yet autoregressive decoding imposes a fundamental latency bottleneck that limits real-time applications such as embodied intelligence, game AI, and interactive avatars (e.g., 10 Hz control frequency). We observe that function calling differs fundamentally from free-form text generation: structured outputs exhibit substantial token redundancy (delimiters, parameter names), and arguments exhibit weak causal dependencies. Crucially, these two properties must be exploited jointly to achieve real-time performance. We present SimpleTool, which introduces special tokens that serve a dual role: compressing low-entropy tokens (4-6x reduction) while acting as mode selectors that enable independent parallel generation of function name and arguments. This synergistic design achieves 3-6x end-to-end speedup (up to 9.6x) with only +8.2% parallelization overhead. Experiments on five benchmarks across Qwen-series models (0.5B-14B) demonstrate substantial speedup while maintaining competitive or improved accuracy. On Mobile Actions, ST-Qwen-0.5B outperforms Google's FunctionGemma in both accuracy and latency consistency. With quantization on consumer-grade GPU, SimpleTool achieves 61.2ms P50 latency, enabling 16 Hz real-time control at 4B model scale, bridging the gap between LLM function calling and latency-critical real-world deployment.
Abstract (translated)
基于大型语言模型（LLM）的功能调用使智能代理能够与外部工具和环境交互，但自回归解码的基本延迟瓶颈限制了实时应用，例如具身智能、游戏AI和互动化身（如10 Hz控制频率）。我们观察到，功能调用在本质上不同于自由形式的文本生成：结构化的输出表现出大量的令牌冗余（分隔符、参数名），并且参数之间表现出弱因果依赖关系。至关重要的是，必须同时利用这两个特性以实现实时性能。 我们介绍了SimpleTool，它引入了特殊令牌，这些令牌具有双重作用：压缩低熵令牌（减少4-6倍）的同时作为模式选择器使功能名称和参数的独立并行生成成为可能。这种协同设计实现了3-6倍的整体加速（最高可达9.6倍），仅增加了8.2%的并行化开销。我们在五个基准测试中对Qwen系列模型（从0.5B到14B）进行了实验，证明了在保持竞争力甚至提高准确性的同时获得了显著的速度提升。 在Mobile Actions上，ST-Qwen-0.5B在准确性和延迟一致性方面均优于Google的FunctionGemma。通过在消费级GPU上的量化处理，SimpleTool实现了61.2ms的P50延迟，在4B规模模型下达到了每秒16次实时控制的速度，从而弥合了LLM功能调用与延迟敏感的真实世界部署之间的差距。
URL
https://arxiv.org/abs/2603.00030
