Abstract
Modular deep learning is the state-of-the-art solution for lifting the curse of multilinguality, preventing the impact of negative interference and enabling cross-lingual performance in Multilingual Pre-trained Language Models. However, a trade-off of this approach is the reduction in positive transfer learning from closely related languages. In response, we introduce a novel method called language arithmetic, which enables training-free post-processing to address this limitation. Inspired by the task arithmetic framework, we apply learning via addition to the language adapters, transitioning the framework from a multi-task to a multilingual setup. The effectiveness of the proposed solution is demonstrated on three downstream tasks in a MAD-X-based set of cross-lingual schemes, acting as a post-processing procedure. Language arithmetic consistently improves the baselines with significant gains in the most challenging cases of zero-shot and low-resource applications. Our code and models are available at this https URL .
Abstract (translated)
模块化深度学习是解决多语言问题的最先进解决方案，可以防止负干扰的影响，并实现跨语言性能。然而，这种方法的一个代价是减少了与相关语言的积极迁移。为了应对这个局限性，我们引入了一种名为语言代数的新方法，它允许无训练的后处理来解决这个问题。受到任务代数框架的启发，我们在语言适配器上进行加法训练，将框架从多任务设置转变为多语言环境。所提出解决方案在基于MAD-X的跨语言方案中的三个下游任务上的有效性得到了说明，充当了一个后处理过程。语言代数在最具挑战性的零 shot 和低资源应用中取得了显著的提高。我们的代码和模型可在此处访问：https://url.com/ 。
URL
https://arxiv.org/abs/2404.15737
