Abstract
Conditional human motion synthesis (HMS) aims to generate human motion sequences that conform to specific conditions. Text and audio represent the two predominant modalities employed as HMS control conditions. While existing research has primarily focused on single conditions, the multi-condition human motion synthesis remains underexplored. In this study, we propose a multi-condition HMS framework, termed MCM, based on a dual-branch structure composed of a main branch and a control branch. This framework effectively extends the applicability of the diffusion model, which is initially predicated solely on textual conditions, to auditory conditions. This extension encompasses both music-to-dance and co-speech HMS while preserving the intrinsic quality of motion and the capabilities for semantic association inherent in the original model. Furthermore, we propose the implementation of a Transformer-based diffusion model, designated as MWNet, as the main branch. This model adeptly apprehends the spatial intricacies and inter-joint correlations inherent in motion sequences, facilitated by the integration of multi-wise self-attention modules. Extensive experiments show that our method achieves competitive results in single-condition and multi-condition HMS tasks.
Abstract (translated)
条件人类运动合成（HMS）旨在生成符合特定条件的的人类运动序列。文本和音频是作为HMS控制条件的两种主要模式。虽然现有的研究主要集中在单个条件，但多条件HMS仍然没有被充分探索。在这项研究中，我们提出了一个基于双分支结构的MCM多条件HMS框架。这个框架基于主分支和控制分支的构建，有效地将扩散模型的应用范围从仅基于文本条件的应用扩展到了基于音频条件的应用。这个扩展包括音乐-舞蹈和共同说话HMS，同时保留原始模型的固有质量和语义关联能力。此外，我们还提出了一个基于Transformer的扩散模型， designated为MWNet，作为主分支。这个模型巧妙地捕捉了运动序列中的空间复杂性和关节相关性，通过集成多维自注意力模块大大促进了这一目的。大量实验结果表明，我们的方法在单条件和多条件HMS任务中实现了竞争力的结果。
URL
https://arxiv.org/abs/2404.12886
