Abstract
Human motion generation has advanced rapidly in recent years, yet the critical problem of creating spatially grounded, context-aware gestures has been largely overlooked. Existing models typically specialize either in descriptive motion generation, such as locomotion and object interaction, or in isolated co-speech gesture synthesis aligned with utterance semantics. However, both lines of work often treat motion and environmental grounding separately, limiting advances toward embodied, communicative agents. To address this gap, our work introduces a multimodal dataset and framework for grounded gesture generation, combining two key resources: (1) a synthetic dataset of spatially grounded referential gestures, and (2) MM-Conv, a VR-based dataset capturing two-party dialogues. Together, they provide over 7.7 hours of synchronized motion, speech, and 3D scene information, standardized in the HumanML3D format. Our framework further connects to a physics-based simulator, enabling synthetic data generation and situated evaluation. By bridging gesture modeling and spatial grounding, our contribution establishes a foundation for advancing research in situated gesture generation and grounded multimodal interaction. Project page: this https URL
Abstract (translated)
近年来，人类动作生成技术迅速发展，然而创建空间定位且具有情境感知能力的手势问题却一直被忽视。现有的模型通常专注于描述性运动生成（如行走和物体互动）或与话语语义相匹配的孤立共言语手势合成。但是，这两种方法往往将运动和环境联系分开处理，从而限制了在实体交互代理方面的进展。为了解决这一差距，我们的工作引入了一个多模态数据集和框架，用于生成基于情境的手势，结合两个关键资源：（1）一个包含空间定位指代手势的合成数据集；（2）MM-Conv，这是一个基于VR的数据集，捕捉了双方对话的情景。这两个资源一起提供了超过7.7小时同步的动作、语音以及3D场景信息，并且这些数据被标准化为HumanML3D格式。我们的框架进一步与物理仿真器连接起来，使合成数据生成和情境评估成为可能。通过将手势建模与空间定位相结合，我们建立了研究基于情景的手势生成及多模态交互的基础。 项目页面：[此网址](this https URL)
URL
https://arxiv.org/abs/2507.04522
