Abstract
The gap between speech and text modalities is a major challenge in speech-to-text translation (ST). Different methods have been proposed for reducing this gap, but most of them require architectural changes in ST training. In this work, we propose to mitigate this issue at the pre-training stage, requiring no change in the ST model. First, we show that the connectionist temporal classification (CTC) loss can reduce the modality gap by design. We provide a quantitative comparison with the more common cross-entropy loss, showing that pre-training with CTC consistently achieves better final ST accuracy. Nevertheless, CTC is only a partial solution and thus, in our second contribution, we propose a novel pre-training method combining CTC and optimal transport to further reduce this gap. Our method pre-trains a Siamese-like model composed of two encoders, one for acoustic inputs and the other for textual inputs, such that they produce representations that are close to each other in the Wasserstein space. Extensive experiments on the standard CoVoST-2 and MuST-C datasets show that our pre-training method applied to the vanilla encoder-decoder Transformer achieves state-of-the-art performance under the no-external-data setting, and performs on par with recent strong multi-task learning systems trained with external data. Finally, our method can also be applied on top of these multi-task systems, leading to further improvements for these models.
Abstract (translated)
语音和文本模式之间的差距是语音到文本翻译(ST)中的一个大挑战。已提出多种方法来减少这种差距,但大多数方法在ST训练阶段需要更改架构。在本文中,我们提出在预训练阶段解决这个问题,不需要更改ST模型。首先,我们表明,连接性时间分类(CTC)损失可以设计地减少模式差距。我们提供与更常见的交叉熵损失的定量比较,表明使用CTC预训练 consistently achieves更好的ST准确性。然而,CTC只是 partial 解决方案,因此我们在第二项贡献中提出了一种结合CTC和最优传输的新预训练方法,进一步减少了这种差距。我们的方法预训练了一个由两个编码器组成的斯泰尔齐尔模型,其中一个用于语音输入,另一个用于文本输入,使他们在瓦塞尔stein空间中产生相互接近的表示。在标准CoVoST-2和MuST-C数据集上的广泛实验表明,我们的方法应用于无外部数据设置下的普通编码器解码器Transformer在无外部数据设置下实现了最先进的性能,并与最近的强大的使用外部数据训练的多项任务学习系统相当。最后,我们的方法也可以应用于这些多项任务系统之上,导致对这些模型的进一步改进。
URL
https://arxiv.org/abs/2301.11716
