Abstract
We present a technical tutorial for building enterprise-grade realtime voice agents from first principles. While over 25 open-source speech-to-speech models and numerous voice agent frameworks exist, no single resource explains the complete pipeline from individual components to a working streaming voice agent with function calling capabilities. Through systematic investigation, we find that (1) native speech-to-speech models like Qwen2.5-Omni, while capable of high-quality audio generation, are too slow for realtime interaction ($\sim$13s time-to-first-audio); (2) the industry-standard approach uses a cascaded streaming pipeline: STT $\rightarrow$ LLM $\rightarrow$ TTS, where each component streams its output to the next; and (3) the key to ``realtime'' is not any single fast model but rather \textit{streaming and pipelining} across components. We build a complete voice agent using Deepgram (streaming STT), vLLM-served LLMs with function calling (streaming text generation), and ElevenLabs (streaming TTS), achieving a measured P50 time-to-first-audio of 947ms (best case 729ms) with cloud LLM APIs, and comparable latency with self-hosted vLLM on NVIDIA A10G GPU. We release the full codebase as a tutorial with working, tested code for every component.
Abstract (translated)
我们提供了一个从基础原理构建企业级实时语音代理的技术教程。尽管存在超过25个开源的语音到语音模型和许多语音代理框架，但没有单一资源能够解释从组件到具有功能调用能力的工作流式语音代理的完整流程。通过系统性研究，我们发现： 1. 本机语音转语音模型（如Qwen2.5-Omni）虽然能生成高质量音频，但由于处理速度过慢而不适合实时交互（约需13秒才能产生首个音频）； 2. 行业标准的方法采用级联流式管道：STT → LLM → TTS，其中每个组件将其输出流传输给下一个组件；以及 3. 实现“实时”的关键不在于单一的快速模型，而在于各组件之间的\textit{流处理和流水线}技术。 我们使用Deepgram（流式语音转文本），由vLLM服务的语言模型（具有功能调用的流式文本生成）和ElevenLabs（流式文本到语音转换），构建了一个完整的语音代理。在使用云端语言模型API时，我们的方案实现了中位数为947毫秒、最佳情况为729毫秒的时间至首次音频输出，并且与自托管vLLM在NVIDIA A10G GPU上的延迟相近。 我们将整个代码库作为教程发布，包括每个组件经过测试的工作代码。
URL
https://arxiv.org/abs/2603.05413
