Abstract
Over the past decade, a series of unflagging efforts have been dedicated to developing highly expressive and controllable text-to-speech (TTS) systems. In general, the holistic TTS comprises two interconnected components: the frontend module and the backend module. The frontend excels in capturing linguistic representations from the raw text input, while the backend module converts linguistic cues to speech. The research community has shown growing interest in the study of the frontend component, recognizing its pivotal role in text-to-speech systems, including Text Normalization (TN), Prosody Boundary Prediction (PBP), and Polyphone Disambiguation (PD). Nonetheless, the limitations posed by insufficient annotated textual data and the reliance on homogeneous text signals significantly undermine the effectiveness of its supervised learning. To evade this obstacle, a novel two-stage TTS frontend prediction pipeline, named TAP-FM, is proposed in this paper. Specifically, during the first learning phase, we present a Multi-scale Contrastive Text-audio Pre-training protocol (MC-TAP), which hammers at acquiring richer insights via multi-granularity contrastive pre-training in an unsupervised manner. Instead of mining homogeneous features in prior pre-training approaches, our framework demonstrates the ability to delve deep into both global and local text-audio semantic and acoustic representations. Furthermore, a parallelized TTS frontend model is delicately devised to execute TN, PD, and PBP prediction tasks, respectively in the second stage. Finally, extensive experiments illustrate the superiority of our proposed method, achieving state-of-the-art performance.
Abstract (translated)
在过去的十年里，一系列无休止的努力都致力于开发高度表达性强且可控制性强的文本转语音（TTS）系统。通常，整体TTS由前端模块和后端模块组成。前端模块在捕捉原始文本输入的语义表示方面表现出色，而后端模块将语义线索转换为语音。研究社区越来越关注前端模块，认识到其在文本转语音系统中的关键作用，包括文本标准化（TN）、 prosody 边界预测（PBP）和多普勒消歧（PD）。然而，缺乏注释的文本数据和依赖同质文本信号的局限性，严重削弱了其监督学习的有效性。为了克服这一障碍，本文提出了一个新颖的两阶段TTS前端预测管道，名为TAP-FM。具体来说，在第一学习阶段，我们提出了一个多尺度对比文本-音频预训练协议（MC-TAP），通过无监督方式通过多粒度对比预训练来获取更丰富的见解。我们不再通过先前的预训练方法挖掘同质特征，而我们的框架能够深入研究全局和局部文本-音频语义和听觉表示。此外，我们还专门设计了一个并行的TTS前端模型，分别执行TN、PD和PBP预测任务。最后，大量实验证实了我们所提出方法的优势，实现了最先进的性能。
URL
https://arxiv.org/abs/2404.09192
