Abstract
We present an innovative end-to-end framework for synthesizing semantically meaningful co-speech gestures and deploying them in real-time on a humanoid robot. This system addresses the challenge of creating natural, expressive non-verbal communication for robots by integrating advanced gesture generation techniques with robust physical control. Our core innovation lies in the meticulous integration of a semantics-aware gesture synthesis module, which derives expressive reference motions from speech input by leveraging a generative retrieval mechanism based on large language models (LLMs) and an autoregressive Motion-GPT model. This is coupled with a high-fidelity imitation learning control policy, the MotionTracker, which enables the Unitree G1 humanoid robot to execute these complex motions dynamically and maintain balance. To ensure feasibility, we employ a robust General Motion Retargeting (GMR) method to bridge the embodiment gap between human motion data and the robot platform. Through comprehensive evaluation, we demonstrate that our combined system produces semantically appropriate and rhythmically coherent gestures that are accurately tracked and executed by the physical robot. To our knowledge, this work represents a significant step toward general real-world use by providing a complete pipeline for automatic, semantic-aware, co-speech gesture generation and synchronized real-time physical deployment on a humanoid robot.
Abstract (translated)
我们提出了一种创新的端到端框架，用于合成具有语义意义的手势，并将其实时部署在人形机器人上。该系统通过将先进的手势生成技术与稳健的物理控制相结合，解决了为机器人创建自然、富有表现力的非语言交流这一挑战。我们的核心创新在于精心整合了一个具备语义感知的手势合成模块。该模块利用基于大型语言模型（LLMs）和自回归Motion-GPT模型的生成检索机制，从语音输入中推导出具有表现力的参考动作。这与高保真的模仿学习控制策略——MotionTracker相耦合，使Unitree G1人形机器人能够动态执行这些复杂的动作，并保持平衡。 为了确保可行性，我们采用了一种稳健的一般运动重定位（GMR）方法，以弥合人类动作数据和机器人平台之间的实现差距。通过全面的评估，我们展示了我们的综合系统生成了语义适当且节奏连贯的手势，物理机器人能够准确地追踪并执行这些手势。 据我们所知，这项工作代表了朝着通用现实世界应用迈出的重要一步，它提供了一个完整的自动、语义感知、与言语同步的手势生成管道，并在人形机器人上实现了实时的物理部署。
URL
https://arxiv.org/abs/2512.17183
