Abstract
Human communication is inherently multimodal and social: words, prosody, and body language jointly carry intent. Yet most prior systems model human behavior as a translation task co-speech gesture or text-to-motion that maps a fixed utterance to motion clips-without requiring agentic decision-making about when to move, what to do, or how to adapt across multi-turn dialogue. This leads to brittle timing, weak social grounding, and fragmented stacks where speech, text, and motion are trained or inferred in isolation. We introduce ViBES (Voice in Behavioral Expression and Synchrony), a conversational 3D agent that jointly plans language and movement and executes dialogue-conditioned body actions. Concretely, ViBES is a speech-language-behavior (SLB) model with a mixture-of-modality-experts (MoME) backbone: modality-partitioned transformer experts for speech, facial expression, and body motion. The model processes interleaved multimodal token streams with hard routing by modality (parameters are split per expert), while sharing information through cross-expert attention. By leveraging strong pretrained speech-language models, the agent supports mixed-initiative interaction: users can speak, type, or issue body-action directives mid-conversation, and the system exposes controllable behavior hooks for streaming responses. We further benchmark on multi-turn conversation with automatic metrics of dialogue-motion alignment and behavior quality, and observe consistent gains over strong co-speech and text-to-motion baselines. ViBES goes beyond "speech-conditioned motion generation" toward agentic virtual bodies where language, prosody, and movement are jointly generated, enabling controllable, socially competent 3D interaction. Code and data will be made available at: this http URL
Abstract (translated)
人类交流本质上是多模态和社交性的：语言、音调以及肢体语言共同传递意图。然而，大多数先前的系统将人类行为建模为一种翻译任务——即伴随言语的手势或文本到动作映射，这一过程固定了一个话语与动作片段之间的关系，并不需要有关何时移动、做什么或如何在多轮对话中适应这些方面的问题决策。这导致了刻板的时间处理方式、弱化的社交背景以及碎片化的方法，在这种方法中，语音、文字和动作的训练或推断都是孤立进行的。 我们引入了一种名为ViBES（行为表达与同步中的声音）的会话式3D代理，它能够同时计划语言和行动，并根据对话条件执行身体动作。具体而言，ViBES是一个包含混合模态专家（MoME）骨干架构的语言-语音-行为（SLB）模型：该模型通过针对语音、面部表情以及肢体运动的不同转换器专家进行分模态处理。模型可以依据模态间交错的多模式标记流进行硬路由（每个专家都有独立的参数），并通过跨专家注意力机制共享信息。 利用强大的预训练语言-语音模型，代理支持混合发起互动：用户可以在对话过程中说话、打字或发出身体动作指令，并且系统提供可控制的行为接口用于实时响应。我们进一步在多轮对话上进行基准测试，并使用自动度量标准评估对话-动作对齐及行为质量，观察到与强大的伴随言语和文本到动作基线相比，ViBES有显著改进。 ViBES超越了“语音驱动的动作生成”，朝向能够共同生成语言、音调和运动的代理式虚拟身体发展。这使得可控且社交胜任度高的3D互动成为可能。代码及数据将在以下链接提供：[此链接]（需要将"http URL"替换为实际提供的GitHub或项目页面地址）。
URL
https://arxiv.org/abs/2512.14234
