Abstract
As embodied agents become central to VR, telepresence, and digital human applications, their motion must go beyond speech-aligned gestures: agents should turn toward users, respond to their movement, and maintain natural gaze. Current methods lack this spatial awareness. We close this gap with the first real-time, fully causal method for spatially-aware conversational motion, deployable on a streaming VR headset. Given a user's position and dyadic audio, our approach produces full-body motion that aligns gestures with speech while orienting the agent according to the user. Our architecture combines a causal transformer-based VAE with interleaved latent tokens for streaming inference and a flow matching model conditioned on user trajectory and audio. To support varying gaze preferences, we introduce a gaze scoring mechanism with classifier-free guidance to decouple learning from control: the model captures natural spatial alignment from data, while users can adjust eye contact intensity at inference time. On the Embody 3D dataset, our method achieves state-of-the-art motion quality at over 300 FPS -- 3x faster than non-causal baselines -- while capturing the subtle spatial dynamics of natural conversation. We validate our approach on a live VR system, bringing spatially-aware conversational agents to real-time deployment. Please see this https URL for details.
Abstract (translated)
随着虚拟现实（VR）、远程呈现和数字人类应用的发展，实体代理的角色变得越来越重要。这些代理的动作需要超越与语音同步的手势，它们应该能够面向用户、响应用户的移动，并保持自然的眼神交流。目前的方法缺乏这种空间感知能力。我们填补了这一空白，推出了首个实时的、完全因果关系的空间感知对话动作生成方法，能够在流媒体VR头显上部署。 我们的方法基于给定用户的位置和双向音频信息来生成全身运动，使手势与语音同步的同时根据用户的移动调整代理的方向。我们的架构结合了一个因果Transformer-based变分自编码器（VAE）以及交错的隐变量令牌以支持流式推理，还有一个受控于用户轨迹和音频的数据匹配模型。 为了适应不同的眼神交流偏好，我们引入了一个眼神评分机制，并使用无分类指导来分离学习与控制：模型从数据中捕捉自然的空间对应关系，而用户则可以在推断时调整目光接触的强度。在Embody 3D数据集上，我们的方法实现了超过每秒300帧的速度（比非因果基准快三倍）并捕获了自然对话中的微妙空间动态变化。 我们通过一个实时VR系统验证了这一方案的有效性，将具有空间感知能力的对话代理带到了现实世界的部署中。详情请参阅此[链接](https://example.com)。
URL
https://arxiv.org/abs/2602.18432
