Abstract
Tibetan, a minority language in China, features a highly intricate grammatical structure, characterized by four verb tenses and a tense system with frequent irregularities, contributing to its extensive inflectional diversity. Recently, advances in Large Language Models (LLMs) have transformed the paradigm in many domains. Despite the success in other fields, current LLMs often fall short in catering to the needs of domain experts like Tibetans, and the potential of LLMs for Tibetan culture is under-explored. The intrinsic reasons are the immense and intricate nature of Tibetan culture as well as the necessity for higher granularity and richness in knowledge. Simultaneously, the complexity and uniqueness of its grammatical structure, coupled with its status as a minority ethnic language, contribute to data scarcity, which remains a fundamental challenge. To alleviate these issues, we introduce Llama-Sunshine (Sun-Shine), the first large language model for Tibetan culture, which is expert in various Tibetan language processing tasks. Sun-Shine incorporates state-of-the-art model architectures optimized for Tibetan's linguistic features. We also propose TIB-STC, a comprehensive dataset comprising diverse Tibetan texts such as literature, religious scripts, news, and conversational data, which is also the first large-scale dataset for Tibetan culture. Though comprehensive experiments, Sun-Shine not only demonstrates a higher level of knowledge expertise for Tibetan culture but also gains preliminary embodied intelligence capabilities in Tibetan language processing tasks, like language modeling, text classification, machine translation, and syntactic analysis. Moreover, it excels in low-resource scenarios, showcasing strong generalization capabilities.
Abstract (translated)
翻译成中文如下： 藏语，作为中国的少数民族语言之一，具有极其复杂的语法结构，包括四个时态和一个经常出现不规则变化的时制系统，这导致了其广泛的词形变化多样性。近年来，大型语言模型（LLMs）在许多领域中改变了范式。尽管在其他领域的应用取得了成功，当前的LLM通常无法满足像藏族这样的专业人员的需求，并且这些模型在藏文化的潜力方面尚未得到充分探索。内在原因是藏文化宏大的复杂性和知识上的细致程度和丰富度的要求。同时，其语法结构的独特复杂性以及作为少数民族语言的地位导致了数据稀缺的问题，这仍然是一个根本性的挑战。为了缓解这些问题，我们推出了Llama-Sunshine（Sun-Shine），这是第一个专门针对藏文化的大型语言模型，擅长处理各种藏语任务。Sun-Shine融合了最先进的模型架构，并针对藏语的语言特征进行了优化。此外，我们还提出了TIB-STC，这是一个包含多种藏文文本的综合性数据集，例如文学作品、宗教文献、新闻和对话数据，也是第一个大规模的藏文化数据集。通过全面的实验，Sun-Shine不仅展示了对藏文化的更高知识专业度，并且在语言建模、文本分类、机器翻译及句法分析等藏语处理任务中获得了初步的实体智能能力。此外，在资源匮乏的情况下，它也表现出强大的泛化能力。
URL
https://arxiv.org/abs/2503.18288
