Abstract
Customizable multilingual zero-shot singing voice synthesis (SVS) has various potential applications in music composition and short video dubbing. However, existing SVS models overly depend on phoneme and note boundary annotations, limiting their robustness in zero-shot scenarios and producing poor transitions between phonemes and notes. Moreover, they also lack effective multi-level style control via diverse prompts. To overcome these challenges, we introduce TCSinger 2, a multi-task multilingual zero-shot SVS model with style transfer and style control based on various prompts. TCSinger 2 mainly includes three key modules: 1) Blurred Boundary Content (BBC) Encoder, predicts duration, extends content embedding, and applies masking to the boundaries to enable smooth transitions. 2) Custom Audio Encoder, uses contrastive learning to extract aligned representations from singing, speech, and textual prompts. 3) Flow-based Custom Transformer, leverages Cus-MOE, with F0 supervision, enhancing both the synthesis quality and style modeling of the generated singing voice. Experimental results show that TCSinger 2 outperforms baseline models in both subjective and objective metrics across multiple related tasks.
Abstract (translated)
可定制的多语言零样本歌声合成（SVS）在音乐创作和短视频配音中具有多种潜在应用。然而，现有的SVS模型过于依赖音素和音符边界的标注，这限制了它们在零样本场景中的鲁棒性，并且导致音素和音符之间的过渡效果不佳。此外，这些模型还缺乏通过多样化提示进行有效的多层次风格控制的能力。 为了解决这些问题，我们引入了TCSinger 2，这是一个基于多种提示的多任务、跨语言零样本歌声合成模型，具备风格转换和风格控制功能。TCSinger 2主要包括三个关键模块： 1. **模糊边界内容（BBC）编码器**：预测时长，扩展内容嵌入，并在边界的上下文应用掩码以实现平滑过渡。 2. **定制音频编码器**：利用对比学习从歌唱、言语和文本提示中提取对齐的表示，增强模型的学习能力与泛化性。 3. **基于流的自定义变压器**：采用Cus-MOE，并通过基频（F0）监督来提高生成歌声的质量和风格建模能力。 实验结果显示，在多个相关任务上，TCSinger 2在主观和客观指标方面均优于基准模型。
URL
https://arxiv.org/abs/2505.14910
