Abstract
We propose Expotion (Facial Expression and Motion Control for Multimodal Music Generation), a generative model leveraging multimodal visual controls - specifically, human facial expressions and upper-body motion - as well as text prompts to produce expressive and temporally accurate music. We adopt parameter-efficient fine-tuning (PEFT) on the pretrained text-to-music generation model, enabling fine-grained adaptation to the multimodal controls using a small dataset. To ensure precise synchronization between video and music, we introduce a temporal smoothing strategy to align multiple modalities. Experiments demonstrate that integrating visual features alongside textual descriptions enhances the overall quality of generated music in terms of musicality, creativity, beat-tempo consistency, temporal alignment with the video, and text adherence, surpassing both proposed baselines and existing state-of-the-art video-to-music generation models. Additionally, we introduce a novel dataset consisting of 7 hours of synchronized video recordings capturing expressive facial and upper-body gestures aligned with corresponding music, providing significant potential for future research in multimodal and interactive music generation.
Abstract (translated)
我们提出了一种名为Expotion（面部表情和动作控制的多模态音乐生成）的生成模型，该模型利用多种视觉控制手段——具体为人类面部表情及上半身动作，并结合文本提示来生成具有表现力且时间准确的音乐。我们在预训练的文本到音乐生成模型基础上采用参数高效的微调方法（PEFT），使得在使用小规模数据集的情况下也能进行细致调整以适应多模态控制。为了确保视频与音乐之间的时间同步，我们引入了一种时序平滑策略来对齐多种模式。 实验结果表明，将视觉特征与文本描述相结合能够显著提升生成音乐的整体质量，在音乐性、创意性、节奏一致性、时间上的视频同步以及遵循文本的程度方面均超越了现有的基线模型和最先进的视频到音乐的生成模型。此外，我们还引入了一个新的数据集，包含7小时的同步视频记录，这些记录捕捉了具有表现力的面部和上半身动作，并且与对应的音乐相匹配。这个数据集为未来多模态和交互式音乐生成的研究提供了巨大的潜力。
URL
https://arxiv.org/abs/2507.04955
