Abstract
Low-resource languages such as Sinhala are often overlooked by open-source Large Language Models (LLMs). In this research, we extend an existing multilingual LLM (Llama-3-8B) to better serve Sinhala. We enhance the LLM tokenizer with Sinhala specific vocabulary and perform continual pre-training on a cleaned 10 million Sinhala corpus, resulting in the SinLlama model. This is the very first decoder-based open-source LLM with explicit Sinhala support. When SinLlama was instruction fine-tuned for three text classification tasks, it outperformed base and instruct variants of Llama-3-8B by a significant margin.
Abstract (translated)
资源较少的语言，如僧伽罗语，在开源大型语言模型（LLM）中往往被忽视。在本研究中，我们将现有的多语言LLM（Llama-3-8B）扩展，以更好地服务于僧伽罗语。我们通过增加针对僧伽罗语的特定词汇来增强LLM分词器，并使用经过清洗的1000万条僧伽罗语文本进行了持续预训练，从而生成了SinLlama模型。这是首个具有明确僧伽罗语支持的基于解码开源大型语言模型。在对三个文本分类任务进行指令微调时，SinLlama的表现远远超过了基础版本和指令变体的Llama-3-8B模型。
URL
https://arxiv.org/abs/2508.09115
