Abstract
Automatic Speech Understanding (ASU) aims at human-like speech interpretation, providing nuanced intent, emotion, sentiment, and content understanding from speech and language (text) content conveyed in speech. Typically, training a robust ASU model relies heavily on acquiring large-scale, high-quality speech and associated transcriptions. However, it is often challenging to collect or use speech data for training ASU due to concerns such as privacy. To approach this setting of enabling ASU when speech (audio) modality is missing, we propose TI-ASU, using a pre-trained text-to-speech model to impute the missing speech. We report extensive experiments evaluating TI-ASU on various missing scales, both multi- and single-modality settings, and the use of LLMs. Our findings show that TI-ASU yields substantial benefits to improve ASU in scenarios where even up to 95% of training speech is missing. Moreover, we show that TI-ASU is adaptive to dropout training, improving model robustness in addressing missing speech during inference.
Abstract (translated)
自动语音理解（ASU）旨在实现人类式的语音理解，提供细微的意图、情感、情感和内容理解，来自语音中传达的语言（文本）内容。通常，训练一个稳健的ASU模型依赖于获取大量高质量的语音及其转录。然而，由于隐私等 concerns，收集或使用语音数据进行ASU训练往往具有挑战性。为了在语音（音频）模式缺失时实现ASU，我们提出了TI-ASU，使用预训练的文本转语音模型进行填补。我们在各种缺失 scale（多模态和单模态）上进行了广泛的实验评估，以及使用LLMs。我们的发现表明，在训练语音甚至缺失至95%时，TI-ASU能够显著提高ASU性能。此外，我们还证明了TI-ASU对缺失语音的适应性，从而提高模型在推理过程中的鲁棒性。
URL
https://arxiv.org/abs/2404.17983
