Abstract
The multilingual neural machine translation (MNMT) enables arbitrary translations across multiple languages by training a model with limited parameters using parallel data only. However, the performance of such MNMT models still lags behind that of large language models (LLMs), limiting their practicality. In this work, we address this limitation by introducing registering to achieve the new state-of-the-art of decoder-only MNMT models. Specifically, we insert a set of artificial tokens specifying the target language, called registers, into the input sequence between the source and target tokens. By modifying the attention mask, the target token generation only pays attention to the activation of registers, representing the source tokens in the target language space. Experiments on EC-40, a large-scale benchmark, show that our method outperforms related methods driven by optimizing multilingual representations. We further scale up and collect 9.3 billion sentence pairs across 24 languages from public datasets to pre-train two models, namely MITRE (multilingual translation with registers). One of them, MITRE-913M, outperforms NLLB-3.3B, achieves comparable performance with commercial LLMs, and shows strong adaptability in fine-tuning. Finally, we open-source our models to facilitate further research and development in MNMT: this https URL.
Abstract (translated)
多语言神经机器翻译（MNMT）通过仅使用平行数据训练具有有限参数的模型，实现了任意多个语言之间的翻译。然而，这种MNMT模型的性能仍然落后于大型语言模型（LLMs），这限制了其实际应用价值。在本项工作中，我们通过引入注册机制来解决这一局限性，并达到了解码器独立MNMT模型的新技术水平。具体来说，在输入序列中，我们将一组指定目标语言的人工标记——称为注册——插入到源语言和目标语言令牌之间。通过对注意力掩码进行修改，目标令牌生成仅关注激活的注册标记，这代表了在目标语言空间中的源语言令牌。 我们在EC-40这一大规模基准测试上的实验表明，我们的方法优于基于优化多语言表示的相关方法。我们进一步扩大规模，在公共数据集中收集了跨越24种语言共计93亿个句子对，并使用这些数据预训练了两个模型，即MITRE（带有注册的多语言翻译）。其中的一个模型MITRE-913M超越了NLLB-3.3B，实现了与商业LLMs相当的表现，并在微调中表现出强大的适应性。最后，我们开源我们的模型，以促进MNMT领域的进一步研究和发展：[此链接](请将此处的占位符替换为实际的URL)。
URL
https://arxiv.org/abs/2501.02979
