Abstract
Neural networks have demonstrated significant advancements in Neural Machine Translation (NMT) compared to conventional phrase-based approaches. However, Multilingual Neural Machine Translation (MNMT) in extremely low-resource settings remains underexplored. This research investigates how knowledge transfer across languages can enhance MNMT in such scenarios. Using the Tatoeba translation challenge dataset from Helsinki NLP, we perform English-German, English-French, and English-Spanish translations, leveraging minimal parallel data to establish cross-lingual mappings. Unlike conventional methods relying on extensive pre-training for specific language pairs, we pre-train our model on English-English translations, setting English as the source language for all tasks. The model is fine-tuned on target language pairs using joint multi-task and sequential transfer learning strategies. Our work addresses three key questions: (1) How can knowledge transfer across languages improve MNMT in extremely low-resource scenarios? (2) How does pruning neuron knowledge affect model generalization, robustness, and catastrophic forgetting? (3) How can TX-Ray interpret and quantify knowledge transfer in trained models? Evaluation using BLEU-4 scores demonstrates that sequential transfer learning outperforms baselines on a 40k parallel sentence corpus, showcasing its efficacy. However, pruning neuron knowledge degrades performance, increases catastrophic forgetting, and fails to improve robustness or generalization. Our findings provide valuable insights into the potential and limitations of knowledge transfer and pruning in MNMT for extremely low-resource settings.
Abstract (translated)
神经网络在神经机器翻译（NMT）中与传统的基于短语的方法相比展现出了显著的进步。然而，在极低资源设置下的多语言神经机器翻译（MNMT）仍然研究不足。这项研究调查了跨语言的知识迁移如何能够改善在这种情景下的MNMT。我们使用来自Helsinki NLP的Tatoeba翻译挑战数据集，进行了英语-德语、英语-法语和英语-西班牙语的翻译工作，利用最少的平行数据建立了跨语言映射。与传统的依赖于特定语言对大量预训练的方法不同，我们在英语-英语的翻译上进行模型的预训练，并将英语设定为所有任务的源语言。该模型通过联合多任务学习和顺序迁移学习策略，在目标语言对上进行了微调。我们的工作回答了三个关键问题：（1）跨语言的知识转移如何改善极低资源场景下的MNMT？（2）剪枝神经元知识如何影响模型的泛化能力、鲁棒性和灾难性遗忘？（3）TX-Ray如何解释和量化训练模型中的知识迁移？通过使用BLEU-4评分进行评估，结果显示顺序迁移学习在包含40k平行句子的数据集上优于基线，展示了其有效性。然而，剪枝神经元知识会降低性能、增加灾难性遗忘，并未能提高鲁棒性和泛化能力。我们的发现为极低资源设置下的MNMT中知识转移和剪枝的潜力与局限提供了有价值的见解。
URL
https://arxiv.org/abs/2412.13881
