Abstract
Singing voice synthesis (SVS) aims to generate expressive and high-quality vocals from musical scores, requiring precise modeling of pitch, duration, and articulation. While diffusion-based models have achieved remarkable success in image and video generation, their application to SVS remains challenging due to the complex acoustic and musical characteristics of singing, often resulting in artifacts that degrade naturalness. In this work, we propose SmoothSinger, a conditional diffusion model designed to synthesize high quality and natural singing voices. Unlike prior methods that depend on vocoders as a final stage and often introduce distortion, SmoothSinger refines low-quality synthesized audio directly in a unified framework, mitigating the degradation associated with two-stage pipelines. The model adopts a reference-guided dual-branch architecture, using low-quality audio from any baseline system as a reference to guide the denoising process, enabling more expressive and context-aware synthesis. Furthermore, it enhances the conventional U-Net with a parallel low-frequency upsampling path, allowing the model to better capture pitch contours and long term spectral dependencies. To improve alignment during training, we replace reference audio with degraded ground truth audio, addressing temporal mismatch between reference and target signals. Experiments on the Opencpop dataset, a large-scale Chinese singing corpus, demonstrate that SmoothSinger achieves state-of-the-art results in both objective and subjective evaluations. Extensive ablation studies confirm its effectiveness in reducing artifacts and improving the naturalness of synthesized voices.
Abstract (translated)
歌声合成（SVS）的目标是从乐谱生成具有表现力和高质量的声乐，这需要对音高、时长和发音进行精确建模。虽然基于扩散的方法在图像和视频生成方面取得了显著的成功，但由于歌唱中复杂的声学和音乐特性，这些方法在SVS中的应用仍然面临挑战，常常导致降低自然度的声音伪影。为此，我们提出了SmoothSinger，这是一种条件式扩散模型，旨在合成高质量且自然的歌声。与以往依赖于编码器作为最终阶段的方法不同，并且这些方法往往会引入失真，SmoothSinger直接在一个统一框架中对低质量的合成音频进行精炼，从而减轻了两阶段流水线所带来的降质问题。 该模型采用了参考引导式双分支架构，使用任何基准系统生成的低质量音频作为参考来指导去噪过程，这使得更具有表现力和上下文感知的合成成为可能。此外，它通过添加一个并行的低频上采样路径增强了传统的U-Net结构，从而使模型能够更好地捕捉音高轮廓和长期频谱依赖关系。 为了在训练过程中改善对齐情况，我们用退化的真实音频替换了参考音频，解决了参考信号与目标信号之间的时序不匹配问题。在大规模中文歌唱语料库Opencpop数据集上的实验表明，SmoothSinger在客观评价和主观评价中均达到了当前最佳水平。广泛的消融研究表明，它在减少伪影和提升合成声音的自然度方面非常有效。
URL
https://arxiv.org/abs/2506.21478
