Abstract
Humans intuitively move to sound, but current humanoid robots lack expressive improvisational capabilities, confined to predefined motions or sparse commands. Generating motion from audio and then retargeting it to robots relies on explicit motion reconstruction, leading to cascaded errors, high latency, and disjointed acoustic-actuation mapping. We propose RoboPerform, the first unified audio-to-locomotion framework that can directly generate music-driven dance and speech-driven co-speech gestures from audio. Guided by the core principle of "motion = content + style", the framework treats audio as implicit style signals and eliminates the need for explicit motion reconstruction. RoboPerform integrates a ResMoE teacher policy for adapting to diverse motion patterns and a diffusion-based student policy for audio style injection. This retargeting-free design ensures low latency and high fidelity. Experimental validation shows that RoboPerform achieves promising results in physical plausibility and audio alignment, successfully transforming robots into responsive performers capable of reacting to audio.
Abstract (translated)
人类天生会随着声音移动，但目前的人形机器人缺乏表达性和即兴表演能力，只能执行预定义的动作或少量命令。从音频生成动作并将其重新定位到机器人上通常依赖于显式的运动重构，这会导致级联误差、高延迟以及声学与驱动之间的映射不连续问题。 我们提出了RoboPerform，这是一个首个统一的音频至行走框架，可以直接根据音乐生成舞蹈，并且可以根据语音生成伴随手势。此框架基于“动作=内容+风格”的核心原则运作，将音频视为隐式的风格信号，从而消除了显式运动重构的需求。RoboPerform整合了一个ResMoE教师策略以适应各种运动模式和一个基于扩散的学生策略来注入音频风格信息。这种无需重新定位的设计确保了低延迟和高保真度。 实验验证显示，RoboPerform在物理真实性和声学同步方面取得了令人满意的结果，成功地将机器人转变成能够响应声音的互动表演者。
URL
https://arxiv.org/abs/2512.23650
