Abstract
Large Language Models (LLMs) have demonstrated remarkable multilingual capabilities, yet challenges persist in adapting these models for low-resource languages. In this study, we investigate the effects of Low-Rank Adaptation (LoRA) Parameter-Efficient Fine-Tuning (PEFT) on multilingual Gemma models for Marathi, a language with limited resources. Using a translated Alpaca dataset with 52,000 instruction-response pairs, our findings reveal that while evaluation metrics often show a performance decline post-fine-tuning, manual assessments frequently suggest that the fine-tuned models outperform their original counterparts. The observations indicate improvements in target language generation capabilities but a reduction in reasoning abilities following language adaptation. These results underscore the need for improved evaluation methodologies and the creation of high-quality native datasets to accurately assess language-specific model performance in low-resource settings.
Abstract (translated)
大型语言模型（LLMs）展现了显著的多语言能力，但在适应低资源语言方面仍存在挑战。在本研究中，我们探讨了低秩适应（LoRA）参数高效微调（PEFT）技术对多种Gemma模型在马拉地语中的影响，这是一种资源有限的语言。使用包含52,000条指令-响应对的翻译Alpaca数据集，我们的研究表明：尽管评估指标常显示微调后性能下降，但手动评估通常表明微调后的模型优于原始模型。观察结果显示出目标语言生成能力的提升，但在适应新语言之后推理能力有所减弱。这些发现强调了改进评估方法和创建高质量本土数据集的必要性，以准确评估低资源环境中的特定语言模型表现。
URL
https://arxiv.org/abs/2411.18571
