Abstract
Singing Voice Synthesis (SVS) remains constrained in practical deployment due to its strong dependence on accurate phoneme-level alignment and manually annotated melody contours, requirements that are resource-intensive and hinder scalability. To overcome these limitations, we propose a melody-driven SVS framework capable of synthesizing arbitrary lyrics following any reference melody, without relying on phoneme-level alignment. Our method builds on a Diffusion Transformer (DiT) architecture, enhanced with a dedicated melody extraction module that derives melody representations directly from reference audio. To ensure robust melody encoding, we employ a teacher model to guide the optimization of the melody extractor, alongside an implicit alignment mechanism that enforces similarity distribution constraints for improved melodic stability and coherence. Additionally, we refine duration modeling using weakly annotated song data and introduce a Flow-GRPO reinforcement learning strategy with a multi-objective reward function to jointly enhance pronunciation clarity and melodic fidelity. Experiments show that our model achieves superior performance over existing approaches in both objective measures and subjective listening tests, especially in zero-shot and lyric adaptation settings, while maintaining high audio quality without manual annotation. This work offers a practical and scalable solution for advancing data-efficient singing voice synthesis. To support reproducibility, we release our inference code and model checkpoints.
Abstract (translated)
歌声合成（SVS）由于其对准确的音素级对齐和人工标注旋律轮廓的高度依赖，仍然在实际部署中受到限制。这些需求耗费资源且阻碍了可扩展性。为克服这些局限性，我们提出了一种基于旋律驱动的SVS框架，该框架能够在没有音素级对齐的情况下根据任何参考旋律合成任意歌词。我们的方法建立在一个改进的Diffusion Transformer（DiT）架构之上，并加入了一个专门用于从参考音频中直接提取旋律表示的模块。为了确保稳定的旋律编码，我们使用一个教师模型来指导旋律抽取器的优化过程，并且引入了一种隐式对齐机制，通过施加相似度分布约束来提高旋律稳定性和连贯性。 此外，我们利用弱标注歌曲数据改进持续时间建模，并采用具有多目标奖励函数的Flow-GRPO强化学习策略，以同时提升发音清晰度和旋律保真度。实验表明，在客观衡量标准及主观听觉测试中，我们的模型在零样本设置和歌词适应环境中均优于现有方法，并且在没有人工注释的情况下仍能保持高质量音频输出。 本研究为推进数据高效歌声合成提供了一个实用而可扩展的解决方案。为了支持再现性，我们发布了推理代码和模型检查点。
URL
https://arxiv.org/abs/2512.04779
