Abstract
We propose MuseControlLite, a lightweight mechanism designed to fine-tune text-to-music generation models for precise conditioning using various time-varying musical attributes and reference audio signals. The key finding is that positional embeddings, which have been seldom used by text-to-music generation models in the conditioner for text conditions, are critical when the condition of interest is a function of time. Using melody control as an example, our experiments show that simply adding rotary positional embeddings to the decoupled cross-attention layers increases control accuracy from 56.6% to 61.1%, while requiring 6.75 times fewer trainable parameters than state-of-the-art fine-tuning mechanisms, using the same pre-trained diffusion Transformer model of Stable Audio Open. We evaluate various forms of musical attribute control, audio inpainting, and audio outpainting, demonstrating improved controllability over MusicGen-Large and Stable Audio Open ControlNet at a significantly lower fine-tuning cost, with only 85M trainble parameters. Source code, model checkpoints, and demo examples are available at: https: //MuseControlLite.this http URL.
Abstract (translated)
我们提出了一种名为MuseControlLite的轻量级机制，旨在通过各种随时间变化的音乐属性和参考音频信号对文本到音乐生成模型进行精细调优。关键发现是，在条件随时间变化的情况下，尽管文本条件化中很少使用位置嵌入（Positional Embeddings），它们却至关重要。以旋律控制为例，我们的实验表明，只需在解耦的交叉注意力层上添加旋转位置嵌入，就可以将控制精度从56.6%提升到61.1%，同时训练参数仅为最先进微调机制的六分之一，并使用相同的预训练扩散Transformer模型（来自Stable Audio Open）。我们对各种音乐属性控制形式、音频修补和音频扩展进行了评估，证明了与MusicGen-Large和Stable Audio Open ControlNet相比，在显著降低微调成本的情况下，提高了可控性。MuseControlLite仅需85M可训练参数。源代码、模型检查点及演示示例可在以下网址获取：https://MuseControlLite.this http URL. 请注意，实际链接可能需要进一步确认或替换为正确的URL地址。
URL
https://arxiv.org/abs/2506.18729
