Abstract
Embodied Conversational Agents (ECAs) aim to emulate human face-to-face interaction through speech, gestures, and facial expressions. Current large language model (LLM)-based conversational agents lack embodiment and the expressive gestures essential for natural interaction. Existing solutions for ECAs often produce rigid, low-diversity motions, that are unsuitable for human-like interaction. Alternatively, generative methods for co-speech gesture synthesis yield natural body gestures but depend on future speech context and require long run-times. To bridge this gap, we present MIBURI, the first online, causal framework for generating expressive full-body gestures and facial expressions synchronized with real-time spoken dialogue. We employ body-part aware gesture codecs that encode hierarchical motion details into multi-level discrete tokens. These tokens are then autoregressively generated by a two-dimensional causal framework conditioned on LLM-based speech-text embeddings, modeling both temporal dynamics and part-level motion hierarchy in real time. Further, we introduce auxiliary objectives to encourage expressive and diverse gestures while preventing convergence to static poses. Comparative evaluations demonstrate that our causal and real-time approach produces natural and contextually aligned gestures against recent baselines. We urge the reader to explore demo videos on this https URL.
Abstract (translated)
实体对话代理（ECAs）旨在通过语音、手势和面部表情模拟面对面的人类互动。当前基于大型语言模型（LLM）的对话代理缺乏身体表现力以及自然互动所必需的表情动作。现有的ECAs解决方案通常会产生僵硬且多样性较低的动作，不适合模仿人类交互。另一方面，用于同步生成自然肢体动作的方法虽然依赖于未来语境中的言语内容，并能产生自然的身体手势，但需要较长的时间来运行。 为了弥合这一差距，我们介绍了MIBURI，这是一个在线、因果框架，用于实时合成与对话语音相匹配的富有表现力的全身动作和面部表情。我们采用肢体部位感知的手势编解码器，将分层运动细节编码为多级离散令牌，并通过基于LLM的言语-文本嵌入条件，在二维因果框架内自回归生成这些令牌，从而实时建模时间动态和部分级别的运动层次结构。此外，为了鼓励产生富有表现力且多样化的手势并防止收敛到静态姿态，我们引入了辅助目标。 对比评估表明，我们的因果性和实时性方法在最近的基准测试中产生的动作自然且与上下文相关。我们建议读者访问以下链接查看演示视频：[此处插入实际URL]（请根据实际情况提供具体链接）。
URL
https://arxiv.org/abs/2603.03282
