Abstract
Neural Machine Translation (NMT) models can be specialized by domain adaptation, often involving fine-tuning on a dataset of interest. This process risks catastrophic forgetting: rapid loss of generic translation quality. Forgetting has been widely observed, with many mitigation methods proposed. However, the causes of forgetting and the relationship between forgetting and adaptation data are under-explored. This paper takes a novel approach to understanding catastrophic forgetting during NMT adaptation by investigating the impact of the data. We provide a first investigation of what is forgotten, and why. We examine the relationship between forgetting and the in-domain data, and show that the amount and type of forgetting is linked to that data's target vocabulary coverage. Our findings pave the way toward better informed NMT domain adaptation.
Abstract (translated)
神经机器翻译（NMT）模型可以通过领域适应进行专业化，这通常涉及在感兴趣的特定数据集上微调。这一过程存在灾难性遗忘的风险：即通用翻译质量的快速下降。遗忘现象已被广泛观察到，并提出了许多缓解方法。然而，遗忘的原因以及遗忘与适应数据之间的关系尚未得到充分探索。本文采用了一种新颖的方法来理解NMT领域适应期间的灾难性遗忘，通过研究数据的影响来进行这一探索。我们首次调查了什么被遗忘了，为什么会发生这种情况。我们探讨了遗忘和特定领域数据之间的关系，并表明遗忘的数量和类型与其目标词汇覆盖范围有关。我们的发现为更好地指导NMT领域的适应铺平了道路。
URL
https://arxiv.org/abs/2412.17537
