Abstract
Nepali, a low-resource language spoken by over 32 million people, continues to face challenges in natural language processing (NLP) due to its complex grammar, agglutinative morphology, and limited availability of high-quality corpora. Most efforts to date have centered on basic encoder architectures; they remain insufficient for Nepali-specific text generation. This study presents a GPT-2-based Nepali language model trained using several training strategies inspired by GPT-3, including optimized learning rate schedules, batch scaling, and architectural refinements. A custom 16k Byte-Pair Encoding (BPE) tokenizer was trained exclusively on Nepali text to ensure more consistent segmentation and improved input representation. The model was pretrained on a combined dataset comprising a 10.75GB cleaned NepBERTa corpus and additional web-scraped Nepali news articles. FlashAttention was integrated to reduce memory usage and stabilize training. After two epochs, the model achieved a training loss of 3.168177, a validation loss of 3.081982, and a final perplexity of 21.80, demonstrating its capability to generate coherent Nepali news-style text.
Abstract (translated)
尼泊尔语是一种资源较少的语言，全世界有超过三千万人使用。由于其复杂的语法、黏着形态学以及高质量语料库的稀缺性，自然语言处理（NLP）领域在处理尼泊尔语时仍面临诸多挑战。迄今为止，大多数努力集中在基本编码架构上；然而，这些方法对于特定于尼泊尔语的文本生成来说依然不够充分。本研究提出了一种基于GPT-2的尼泊尔语语言模型，并采用了几种受GPT-3启发的训练策略进行训练，包括优化的学习率计划、批量缩放以及架构改进。为了确保更一致的分词和改善输入表示，我们专门为尼泊尔文本训练了一个定制的16k字节对编码（BPE）词元化器。模型在包含经清理后的NepBERTa语料库（大小为10.75GB）以及额外抓取的尼泊尔新闻文章所组成的组合数据集上进行了预训练。此外，还整合了FlashAttention技术以减少内存使用量并稳定训练过程。经过两轮训练后，模型达到了3.168177的训练损失、3.081982的验证损失以及最终困惑度为21.80的表现，证明其能够生成连贯的尼泊尔新闻风格文本。
URL
https://arxiv.org/abs/2512.14585
