Abstract
In this study, we revisit the commonly-cited off-target issue in multilingual neural machine translation (MNMT). By carefully designing experiments on different MNMT scenarios and models, we attribute the off-target issue to the overfitting of the shortcuts of (non-centric, centric) language mappings. Specifically, the learned shortcuts biases MNMT to mistakenly translate non-centric languages into the centric language instead of the expected non-centric language for zero-shot translation. Analyses on learning dynamics show that the shortcut learning generally occurs in the later stage of model training, and multilingual pretraining accelerates and aggravates the shortcut learning. Based on these observations, we propose a simple and effective training strategy to eliminate the shortcuts in MNMT models by leveraging the forgetting nature of model training. The only difference from the standard training is that we remove the training instances that may induce the shortcut learning in the later stage of model training. Without introducing any additional data and computational costs, our approach can consistently and significantly improve the zero-shot translation performance by alleviating the shortcut learning for different MNMT models and benchmarks.
Abstract (translated)
在这项研究中，我们重新审视了多语言神经机器翻译（MNMT）中经常提及的非目标问题。通过在不同的MNMT场景和模型上精心设计实验，我们将非目标问题归因于（非中心语言，中心语言）映射快捷方式的过拟合。具体来说，学习到的快捷方式使MNMT偏向错误地将非中心语言翻译为中心语言，而不是预期中的零样本翻译的非中心语言。对学习动态的分析表明，快捷方式的学习通常发生在模型训练的后期阶段，并且多语言预训练加速和加剧了快捷方式的学习。基于这些观察结果，我们提出了一种简单有效的训练策略，通过利用模型训练中的遗忘特性来消除MNMT模型中的快捷方式。与标准训练相比，唯一的不同是我们移除了可能导致快捷方式学习的训练实例，在训练后期阶段。无需引入任何额外的数据或计算成本，我们的方法可以持续显著地改善不同MNMT模型和基准上的零样本翻译性能，通过缓解快捷方式的学习达成此效果。
URL
https://arxiv.org/abs/2411.10581
