Abstract
End-to-end speech-to-text translation typically suffers from the scarcity of paired speech-text data. One way to overcome this shortcoming is to utilize the bitext data from the Machine Translation (MT) task and perform Multi-Task Learning (MTL). In this paper, we formulate MTL from a regularization perspective and explore how sequences can be regularized within and across modalities. By thoroughly investigating the effect of consistency regularization (different modality) and R-drop (same modality), we show how they respectively contribute to the total regularization. We also demonstrate that the coefficient of MT loss serves as another source of regularization in the MTL setting. With these three sources of regularization, we introduce the optimal regularization contour in the high-dimensional space, called the regularization horizon. Experiments show that tuning the hyperparameters within the regularization horizon achieves near state-of-the-art performance on the MuST-C dataset.
Abstract (translated)
端到端的语音转文本翻译通常会受到成对的语音-文本数据稀缺性的限制。克服这一问题的一种方法是利用机器翻译（MT）任务中的双语数据，并进行多任务学习（MTL）。在这篇论文中，我们从正则化角度出发来构建多任务学习框架，并探讨了如何在不同模态内和跨模态之间对序列进行正则化。通过深入研究一致性正则化（不同模态）和R-drop（相同模态）的效果，展示了它们各自对于总正则化的贡献。我们还表明，在多任务设置中，机器翻译损失的系数充当另一种正则化来源。利用这三种正则化来源，我们在高维空间中引入了一种名为正则化地平线的最佳正则化轮廓。实验结果表明，在MuST-C数据集上调整该正则化地平线内的超参数可以实现接近最先进的性能。
URL
https://arxiv.org/abs/2509.09701
