Abstract
Sarcastic speech synthesis, which involves generating speech that effectively conveys sarcasm, is essential for enhancing natural interactions in applications such as entertainment and human-computer interaction. However, synthesizing sarcastic speech remains a challenge due to the nuanced prosody that characterizes sarcasm, as well as the limited availability of annotated sarcastic speech data. To address these challenges, this study introduces a novel approach that integrates feedback loss from a bi-modal sarcasm detection model into the TTS training process, enhancing the model's ability to capture and convey sarcasm. In addition, by leveraging transfer learning, a speech synthesis model pre-trained on read speech undergoes a two-stage fine-tuning process. First, it is fine-tuned on a diverse dataset encompassing various speech styles, including sarcastic speech. In the second stage, the model is further refined using a dataset focused specifically on sarcastic speech, enhancing its ability to generate sarcasm-aware speech. Objective and subjective evaluations demonstrate that our proposed methods improve the quality, naturalness, and sarcasm-awareness of synthesized speech.
Abstract (translated)
讽刺语调的语音合成，即生成能够有效传达讽刺意味的语音，在娱乐和人机交互应用中提高自然互动体验方面至关重要。然而，由于讽刺表达具有复杂的语调特点以及可用的标注讽刺语料数据有限，因此在实际操作中实现这一目标仍面临诸多挑战。 为了应对这些挑战，本研究提出了一种创新方法，该方法将双模态讽刺检测模型提供的反馈损失融入到文本转语音（TTS）训练过程中。通过这种方式，可以增强模型捕捉和传达讽刺信息的能力。此外，在进行迁移学习的基础上，先对大量朗读语料预训练的语音合成模型进行了两阶段微调处理：第一阶段基于涵盖各种语言风格（包括讽刺表达）的数据集；第二阶段则专门使用聚焦于讽刺表达的数据集进一步优化模型，使其能够生成带有讽刺意味的语音。 通过客观和主观评估，我们证明了所提出的方法可以提高合成语音的质量、自然度以及其对讽刺语境的理解力。
URL
https://arxiv.org/abs/2508.13028
