Abstract
The accompanying actions and gestures in dialogue are often closely linked to interactions with the environment, such as looking toward the interlocutor or using gestures to point to the described target at appropriate moments. Speech and semantics guide the production of gestures by determining their timing (WHEN) and style (HOW), while the spatial locations of interactive objects dictate their directional execution (WHERE). Existing approaches either rely solely on descriptive language to generate motions or utilize audio to produce non-interactive gestures, thereby lacking the characterization of interactive timing and spatial intent. This significantly limits the applicability of conversational gesture generation, whether in robotics or in the fields of game and animation production. To address this gap, we present a full-stack solution. We first established a unique data collection method to simultaneously capture high-precision human motion and spatial intent. We then developed a generation model driven by audio, language, and spatial data, alongside dedicated metrics for evaluating interaction timing and spatial accuracy. Finally, we deployed the solution on a humanoid robot, enabling rich, context-aware physical interactions.
Abstract (translated)
对话中的伴随动作和手势通常与环境互动紧密相关，例如朝对方看或在适当时刻用手势指向所描述的目标。言语和语义通过确定手势的时间（何时）和风格（如何）来指导手势的产生，而交互对象的空间位置则决定了其执行的方向（何处）。现有的方法要么仅依赖于描述性语言生成动作，要么利用音频生成非互动性手势，因此缺乏对互动时机和空间意图的刻画。这大大限制了对话中手势生成的应用范围，无论是机器人领域还是游戏与动画制作行业。 为解决这一问题，我们提出了一整套解决方案。首先，我们建立了一个独特的方法来同时捕捉高质量的人体动作和空间意图数据。接着，开发了一种由音频、语言和空间数据驱动的生成模型，并制定了评估互动时机和空间准确性专门指标。最后，我们将该方案部署在人形机器人上，实现了丰富且具备情境意识的物理交互。
URL
https://arxiv.org/abs/2509.23852
