Abstract
Multilingual neural machine translation has witnessed remarkable progress in recent years. However, the long-tailed distribution of multilingual corpora poses a challenge of Pareto optimization, i.e., optimizing for some languages may come at the cost of degrading the performance of others. Existing balancing training strategies are equivalent to a series of Pareto optimal solutions, which trade off on a Pareto frontier. In this work, we propose a new training framework, Pareto Mutual Distillation (Pareto-MD), towards pushing the Pareto frontier outwards rather than making trade-offs. Specifically, Pareto-MD collaboratively trains two Pareto optimal solutions that favor different languages and allows them to learn from the strengths of each other via knowledge distillation. Furthermore, we introduce a novel strategy to enable stronger communication between Pareto optimal solutions and broaden the applicability of our approach. Experimental results on the widely-used WMT and TED datasets show that our method significantly pushes the Pareto frontier and outperforms baselines by up to +2.46 BLEU.
Abstract (translated)
跨语言神经网络机器翻译在近年来取得了显著进展。然而，多语言 corpora 的长尾分布提出了对 Pareto 优化的挑战，即对某些语言进行优化可能会牺牲其他语言的性能。现有的平衡训练策略等价于一系列 Pareto 最优解决方案，它们在 Pareto Frontier 上进行权衡。在本文中，我们提出了一个新的训练框架，Pareto mutual Distillation(Pareto-MD)，旨在推动 Pareto Frontier 向外扩展，而不是进行权衡。具体来说，Pareto-MD 合作训练两个 Pareto 最优解决方案，它们分别偏好不同的语言，并通过知识蒸馏学习彼此的优点。此外，我们提出了一种新策略，以增强 Pareto 最优解决方案之间的通信并扩大我们方法的适用性。在广泛使用的 WMT 和 TED 数据集上进行了实验，结果显示，我们的方法 significantly 推动了 Pareto Frontier 并比基准方法高出 2.46BLEU。
URL
https://arxiv.org/abs/2305.15718
