Abstract
Whole-body multi-modal human motion generation poses two primary challenges: creating an effective motion generation mechanism and integrating various modalities, such as text, speech, and music, into a cohesive framework. Unlike previous methods that usually employ discrete masked modeling or autoregressive modeling, we develop a continuous masked autoregressive motion transformer, where a causal attention is performed considering the sequential nature within the human motion. Within this transformer, we introduce a gated linear attention and an RMSNorm module, which drive the transformer to pay attention to the key actions and suppress the instability caused by either the abnormal movements or the heterogeneous distributions within multi-modalities. To further enhance both the motion generation and the multimodal generalization, we employ the DiT structure to diffuse the conditions from the transformer towards the targets. To fuse different modalities, AdaLN and cross-attention are leveraged to inject the text, speech, and music signals. Experimental results demonstrate that our framework outperforms previous methods across all modalities, including text-to-motion, speech-to-gesture, and music-to-dance. The code of our method will be made public.
Abstract (translated)
全身多模态人体动作生成面临两个主要挑战：创建有效的运动生成机制以及将各种模式（如文本、语音和音乐）整合到一个连贯的框架中。与以往通常采用离散掩码建模或自回归建模的方法不同，我们开发了一种连续的掩码自回归运动变换器，在该模型中考虑到人体动作中的序列特性而执行因果注意力操作。在这个变换器内，我们引入了门控线性注意机制和RMSNorm模块，这些设计促使变换器关注关键的动作，并抑制由于异常移动或多模态分布差异所导致的不稳定性。 为了进一步增强运动生成和多模式泛化能力，我们采用了DiT结构来从变换器向目标扩散条件。在融合不同模式时，我们利用AdaLN和交叉注意力机制将文本、语音和音乐信号注入到模型中。实验结果表明，我们的框架在所有模态（包括文本到动作、语音到手势以及音乐到舞蹈）上都优于现有方法。 我们的方法代码将在适当的时候公开发布。
URL
https://arxiv.org/abs/2510.14954
