Abstract
We propose a unified framework for Singing Voice Synthesis (SVS) and Conversion (SVC), addressing the limitations of existing approaches in cross-domain SVS/SVC, poor output musicality, and scarcity of singing data. Our framework enables control over multiple aspects, including language content based on lyrics, performance attributes based on a musical score, singing style and vocal techniques based on a selector, and voice identity based on a speech sample. The proposed zero-shot learning paradigm consists of one SVS model and two SVC models, utilizing pre-trained content embeddings and a diffusion-based generator. The proposed framework is also trained on mixed datasets comprising both singing and speech audio, allowing singing voice cloning based on speech reference. Experiments show substantial improvements in timbre similarity and musicality over state-of-the-art baselines, providing insights into other low-data music tasks such as instrumental style transfer. Examples can be found at: this http URL.
Abstract (translated)
我们提出了一种统一的框架，用于歌唱声音合成（SVS）和转换（SVC），以解决现有方法在跨域SVS/SVC、输出音乐性差以及缺乏歌唱数据方面的限制。我们的框架支持对多个方面进行控制，包括基于歌词的语言内容、基于乐谱的表现属性、基于选择器的演唱风格和技术手段，以及基于语音样本的声音身份。 所提出的零样本学习范式包含一个SVS模型和两个SVC模型，并使用预训练的内容嵌入和基于扩散的生成器。我们的框架还利用混合数据集进行训练，该数据集中包括歌唱和讲话音频，从而允许根据讲话参考实现歌唱声音克隆。实验表明，在音色相似度和音乐性方面，与最先进的基准相比有显著改进，这为其他低数据量的音乐任务（如乐器风格转换）提供了见解。 示例可以在以下网址找到：this http URL.
URL
https://arxiv.org/abs/2501.13870
