Abstract
Conventional text-to-speech (TTS) research has predominantly focused on enhancing the quality of synthesized speech for speakers in the training dataset. The challenge of synthesizing lifelike speech for unseen, out-of-dataset speakers, especially those with limited reference data, remains a significant and unresolved problem. While zero-shot or few-shot speaker-adaptive TTS approaches have been explored, they have many limitations. Zero-shot approaches tend to suffer from insufficient generalization performance to reproduce the voice of speakers with heavy accents. While few-shot methods can reproduce highly varying accents, they bring a significant storage burden and the risk of overfitting and catastrophic forgetting. In addition, prior approaches only provide either zero-shot or few-shot adaptation, constraining their utility across varied real-world scenarios with different demands. Besides, most current evaluations of speaker-adaptive TTS are conducted only on datasets of native speakers, inadvertently neglecting a vast portion of non-native speakers with diverse accents. Our proposed framework unifies both zero-shot and few-shot speaker adaptation strategies, which we term as "instant" and "fine-grained" adaptations based on their merits. To alleviate the insufficient generalization performance observed in zero-shot speaker adaptation, we designed two innovative discriminators and introduced a memory mechanism for the speech decoder. To prevent catastrophic forgetting and reduce storage implications for few-shot speaker adaptation, we designed two adapters and a unique adaptation procedure.
Abstract (translated)
传统的文本转语音（TTS）研究主要集中在提高训练数据中合成语音的质量，特别是对于训练数据中的说话人。为训练数据中的未见、无数据来源的说话人合成自然流畅的语音仍然是一个显著且未解决的问题。尽管已经探索了零 shot 或少数 shot 的说话人自适应TTS方法，但它们存在许多局限性。零 shot 方法往往不足以复制具有严重口音的说话人的声音，而少数 shot 方法虽然可以复制高度变化的口音，但会带来显著的存储负担和过拟合和灾难性忘记的风险。此外，之前的方法只提供了零 shot 或少数 shot 的适应性，限制了它们在不同现实场景中的使用。除此之外，大多数对说话人自适应TTS的评估仅在本土说话人的数据集上进行，无意中忽略了具有不同口音的非本土说话人。我们提出的框架将零 shot 和少数 shot 的适应性策略统一起来，我们称之为“即兴”和“细粒度”适应性策略，基于它们的优点。为了缓解零 shot 说话人适应性中观察到的不足，我们设计了两个创新的分歧器和引入了语音解码器的记忆机制。为了防止灾难性忘记和减少零 shot 说话人适应性中的存储影响，我们设计了两个适配器和一种独特的适应程序。
URL
https://arxiv.org/abs/2404.18094
