Abstract
As large language models (LLMs) are trained on increasingly diverse and extensive multilingual corpora, they demonstrate cross-lingual transfer capabilities. However, these capabilities often fail to effectively extend to low-resource languages, particularly those utilizing non-Latin scripts. While transliterating low-resource languages into Latin script presents a natural solution, there currently lacks a comprehensive framework for integrating transliteration into LLMs training and deployment. Taking a pragmatic approach, this paper innovatively combines character transliteration with Huffman coding to design a complete transliteration framework. Our proposed framework offers the following advantages: 1) Compression: Reduces storage requirements for low-resource language content, achieving up to 50% reduction in file size and 50-80% reduction in token count. 2) Accuracy: Guarantees 100% lossless conversion from transliterated text back to the source language. 3) Efficiency: Eliminates the need for vocabulary expansion for low-resource languages, improving training and inference efficiency. 4) Scalability: The framework can be extended to other low-resource languages. We validate the effectiveness of our framework across multiple downstream tasks, including text classification, machine reading comprehension, and machine translation. Experimental results demonstrate that our method significantly enhances the model's capability to process low-resource languages while maintaining performance on high-resource languages. Our data and code are publicly available at this https URL.
Abstract (translated)
随着大型语言模型（LLMs）在日益多样化和广泛的多语种语料库上进行训练，它们展示出了跨语言迁移能力。然而，这些能力往往未能有效地扩展到低资源语言中，尤其是那些使用非拉丁字母的语言。虽然将低资源语言转写为拉丁字母提供了一个自然的解决方案，但目前尚缺乏一个全面的框架来将这种转写集成到LLMs的训练和部署过程中。本文采用了一种实用的方法，创新性地结合了字符转写与霍夫曼编码，设计出一套完整的转写框架。 我们提出的框架具有以下优势： 1) 压缩：减少了低资源语言内容所需的存储要求，文件大小可减少高达50%，标记数量可以减少50-80%。 2) 准确性：确保从转写字体文本转换回源语言时无损。 3) 效率：消除了对低资源语言词汇表扩展的需求，提高了训练和推理的效率。 4) 可扩展性：该框架可以应用于其他低资源语言。 我们在多个下游任务中验证了我们框架的有效性，包括文本分类、机器阅读理解以及机器翻译。实验结果表明，我们的方法显著提升了模型处理低资源语言的能力，并且同时保持在高资源语言上的性能不变。我们的数据和代码可在以下网址公开获取：[此URL]（请根据实际情况替换为具体的链接）。
URL
https://arxiv.org/abs/2509.17493
