Abstract
Controlling text-to-speech (TTS) systems to synthesize speech with the prosodic characteristics expected by users has attracted much attention. To achieve controllability, current studies focus on two main directions: (1) using reference speech as prosody prompt to guide speech synthesis, and (2) using natural language descriptions to control the generation process. However, finding reference speech that exactly contains the prosody that users want to synthesize takes a lot of effort. Description-based guidance in TTS systems can only determine the overall prosody, which has difficulty in achieving fine-grained prosody control over the synthesized speech. In this paper, we propose DrawSpeech, a sketch-conditioned diffusion model capable of generating speech based on any prosody sketches drawn by users. Specifically, the prosody sketches are fed to DrawSpeech to provide a rough indication of the expected prosody trends. DrawSpeech then recovers the detailed pitch and energy contours based on the coarse sketches and synthesizes the desired speech. Experimental results show that DrawSpeech can generate speech with a wide variety of prosody and can precisely control the fine-grained prosody in a user-friendly manner. Our implementation and audio samples are publicly available.
Abstract (translated)
控制文本到语音（TTS）系统，使其生成具有用户期望的韵律特征的语音，已经吸引了广泛的关注。为了实现这种可控性，目前的研究主要集中在两个方向：(1) 使用参考语音作为韵律提示来指导语音合成；(2) 使用自然语言描述来控制生成过程。然而，找到包含用户希望合成的特定韵律特性的参考语音需要大量努力。基于描述的方法在TTS系统中只能确定整体的韵律特征，在实现对合成语音细粒度的韵律控制方面存在困难。 在本文中，我们提出了一种名为DrawSpeech的新模型，这是一种可以根据用户绘制的任何韵律草图生成语音的扩散模型。具体来说，用户的韵律草图被输入到DrawSpeech系统中，以提供期望的韵律趋势的一个粗略指示。然后，DrawSpeech根据这些粗糙的草图恢复出详细的音高和能量轮廓，并最终合成所需的语音。 实验结果表明，DrawSpeech可以生成具有广泛不同韵律特征的语音，并且可以通过用户友好的方式精确控制细粒度的韵律特性。我们的实现代码及音频样本都公开可用。
URL
https://arxiv.org/abs/2501.04256
