Abstract
While neural text-to-speech (TTS) has achieved human-like natural synthetic speech, multilingual TTS systems are limited to resource-rich languages due to the need for paired text and studio-quality audio data. This paper proposes a method for zero-shot multilingual TTS using text-only data for the target language. The use of text-only data allows the development of TTS systems for low-resource languages for which only textual resources are available, making TTS accessible to thousands of languages. Inspired by the strong cross-lingual transferability of multilingual language models, our framework first performs masked language model pretraining with multilingual text-only data. Then we train this model with a paired data in a supervised manner, while freezing a language-aware embedding layer. This allows inference even for languages not included in the paired data but present in the text-only data. Evaluation results demonstrate highly intelligible zero-shot TTS with a character error rate of less than 12% for an unseen language. All experiments were conducted using public datasets and the implementation will be made available for reproducibility.
Abstract (translated)
虽然神经网络文本语音(TTS)技术已经实现了类似于自然生成的人类语音,但由于需要对两个文本进行配对以及高质量的音频数据,多语言TTS系统仍然局限于资源丰富的语言。本文提出了一种方法,用于零次配对多语言TTS,使用目标语言的文本数据。使用文本数据可以开发低资源语言的TTS系统,这些语言只有文本资源可用,从而使TTS对数千个语言开放。受到多语言语言模型的强烈跨语言互操作性的影响,我们的框架首先使用多语言文本只数据进行掩码语言模型的预训练。然后,我们使用一对数据进行 supervised 训练,同时冻结一个语言意识的嵌入层。这即使不包括在配对数据中的语言,但在文本只数据中存在也可以进行推断。评估结果显示,零次配对的TTS文本错误率小于12%。所有实验使用公开数据集进行,实现将提供可重复使用。
URL
https://arxiv.org/abs/2301.12596
