Abstract
We tackle the task of long-form music generation--particularly the challenging \textbf{lyrics-to-song} problem--by introducing YuE, a family of open foundation models based on the LLaMA2 architecture. Specifically, YuE scales to trillions of tokens and generates up to five minutes of music while maintaining lyrical alignment, coherent musical structure, and engaging vocal melodies with appropriate accompaniment. It achieves this through (1) track-decoupled next-token prediction to overcome dense mixture signals, (2) structural progressive conditioning for long-context lyrical alignment, and (3) a multitask, multiphase pre-training recipe to converge and generalize. In addition, we redesign the in-context learning technique for music generation, enabling versatile style transfer (e.g., converting Japanese city pop into an English rap while preserving the original accompaniment) and bidirectional generation. Through extensive evaluation, we demonstrate that YuE matches or even surpasses some of the proprietary systems in musicality and vocal agility. In addition, fine-tuning YuE enables additional controls and enhanced support for tail languages. Furthermore, beyond generation, we show that YuE's learned representations can perform well on music understanding tasks, where the results of YuE match or exceed state-of-the-art methods on the MARBLE benchmark. Keywords: lyrics2song, song generation, long-form, foundation model, music generation
Abstract (translated)
我们通过引入基于LLaMA2架构的开放基础模型YuE家族，解决了长篇音乐生成任务——尤其是具有挑战性的“歌词到歌曲”问题。具体来说，YuE可以处理万亿级的token，并且能够生成长达五分钟的音乐同时保持歌词的一致性、连贯的音乐结构和吸引人的声乐旋律以及适当的伴奏。它通过以下三种方式实现了这一目标： 1. 轨道解耦下一次令牌预测：克服密集混合信号。 2. 结构渐进条件设置：实现长时间上下文下的歌词一致性。 3. 多任务、多阶段预训练配方，以达成收敛和泛化。 此外，我们重新设计了音乐生成中的上下文学习技术，使其能够进行多样化的风格转换（例如，将日本城市流行乐转化为英文说唱，同时保留原有的伴奏）以及双向生成。通过广泛的评估，我们展示了YuE在音乐性和声乐敏捷性方面可以与一些专有系统匹敌甚至超越。此外，微调YuE还使额外的控制和对尾部语言的支持得到增强。 除了生成之外，我们还证明了YuE的学习表示还可以很好地应用于音乐理解任务，在MARBLE基准测试中，其结果可与现有最佳方法相匹配或超出。 关键词：歌词转歌曲、歌曲生成、长篇形式、基础模型、音乐生成
URL
https://arxiv.org/abs/2503.08638
