Abstract
This paper introduces OmniMotion-X, a versatile multimodal framework for whole-body human motion generation, leveraging an autoregressive diffusion transformer in a unified sequence-to-sequence manner. OmniMotion-X efficiently supports diverse multimodal tasks, including text-to-motion, music-to-dance, speech-to-gesture, and global spatial-temporal control scenarios (e.g., motion prediction, in-betweening, completion, and joint/trajectory-guided synthesis), as well as flexible combinations of these tasks. Specifically, we propose the use of reference motion as a novel conditioning signal, substantially enhancing the consistency of generated content, style, and temporal dynamics crucial for realistic animations. To handle multimodal conflicts, we introduce a progressive weak-to-strong mixed-condition training strategy. To enable high-quality multimodal training, we construct OmniMoCap-X, the largest unified multimodal motion dataset to date, integrating 28 publicly available MoCap sources across 10 distinct tasks, standardized to the SMPL-X format at 30 fps. To ensure detailed and consistent annotations, we render sequences into videos and use GPT-4o to automatically generate structured and hierarchical captions, capturing both low-level actions and high-level semantics. Extensive experimental evaluations confirm that OmniMotion-X significantly surpasses existing methods, demonstrating state-of-the-art performance across multiple multimodal tasks and enabling the interactive generation of realistic, coherent, and controllable long-duration motions.
Abstract (translated)
本文介绍了OmniMotion-X，这是一个用于全身人类动作生成的多功能多模态框架，在统一的序列到序列方式中利用自回归扩散变压器。OmniMotion-X高效支持各种多模态任务，包括文本到动作、音乐到舞蹈、语音到手势以及全局空间时间控制场景（例如运动预测、插值、完成和关节/轨迹引导合成），并且还支持这些任务之间的灵活组合。 具体而言，我们提出将参考动作作为新颖的条件信号使用，这在很大程度上增强了生成内容的一致性、风格和关键的时间动态特性，这对于现实主义动画至关重要。为了处理多模态冲突，我们引入了一种逐步从弱到强混合条件训练策略。为了实现高质量的多模态训练，我们构建了OmniMoCap-X，这是迄今为止最大的统一多模态运动数据集，整合了28个公开的Motion Capture (MoCap) 数据源，涵盖10项不同的任务，并标准化为SMPL-X格式，以每秒30帧的速度进行。 为了确保详细且一致的注释，我们渲染序列成视频，并使用GPT-4o自动生成结构化和分层标题，捕捉低级动作和高级语义。广泛的实验评估确认OmniMotion-X在多个多模态任务上显著优于现有方法，展示了最先进的性能，并能够生成真实、连贯且可控的长持续时间运动。
URL
https://arxiv.org/abs/2510.19789
