Abstract
Fine-tuning is widely used to tailor large language models for specific tasks such as neural machine translation (NMT). However, leveraging transfer learning is computationally expensive when fine-tuning large multilingual models with billions of parameters, thus creating a barrier to entry for researchers working on low-resource domains such as Irish translation. Parameter-efficient fine-tuning (PEFT) bridges this gap by training on a fraction of the original model parameters, with the Low-Rank Adaptation (LoRA) approach introducing small, trainable adapter layers. We introduce SemiAdapt and SemiLoRA as semi-supervised inference-efficient approaches that strengthen domain adaptation and lead to improved overall performance in NMT. We demonstrate that SemiAdapt can outperform full-domain fine-tuning, while most notably, SemiLoRA can propel PEFT methods to match or even outperform full-model fine-tuning. We further evaluate domain-by-dataset fine-tuning and demonstrate that our embedding-based inference methods perform especially well on larger and noisier corpora. All Irish translation models developed in this work are released as open resources. These methods aim to make high-quality domain adaptation and fine-tuning more accessible to researchers working with low-resource languages.
Abstract (translated)
微调广泛应用于根据特定任务（如神经机器翻译(NMT)）调整大型语言模型。然而，当对具有数十亿参数的多语言大模型进行微调时，利用迁移学习在计算上非常昂贵，这为研究资源匮乏领域的研究人员（例如爱尔兰语翻译）设置了进入障碍。参数高效的微调(PEFT)通过仅训练原始模型部分参数来解决这一问题，并且低秩适应(Low-Rank Adaptation, LoRA)方法引入了较小的可训练适配器层。我们介绍了半监督推理高效的方法SemiAdapt和SemiLoRA，它们加强领域适应并使NMT的整体性能得到提升。我们演示了SemiAdapt可以超越全领域微调，而值得注意的是，SemiLoRA可以使PEFT方法与全模型微调相匹配甚至优于后者。进一步地，我们在按数据集进行领域微调时进行了评估，并展示了我们的基于嵌入的推理方法在更大且噪声更多的语料库上表现尤为出色。本工作中开发的所有爱尔兰翻译模型均以开源资源的形式发布。这些方法旨在使低资源语言的研究人员能够更容易地实现高质量领域的适应和微调。
URL
https://arxiv.org/abs/2510.18725
