Abstract
Adapting pretrained language models to low-resource, morphologically rich languages remains a significant challenge. Existing vocabulary expansion methods typically rely on arbitrarily segmented subword units, resulting in fragmented lexical representations and loss of critical morphological information. To address this limitation, we propose the Lexically Grounded Subword Embedding Initialization (LGSE) framework, which introduces morphologically informed segmentation for initializing embeddings of novel tokens. Instead of using random vectors or arbitrary subwords, LGSE decomposes words into their constituent morphemes and constructs semantically coherent embeddings by averaging pretrained subword or FastText-based morpheme representations. When a token cannot be segmented into meaningful morphemes, its embedding is constructed using character n-gram representations to capture structural information. During Language-Adaptive Pretraining, we apply a regularization term that penalizes large deviations of newly introduced embeddings from their initialized values, preserving alignment with the original pretrained embedding space while enabling adaptation to the target language. To isolate the effect of initialization, we retain the original pre-trained model vocabulary and tokenizer and update only the new embeddings during adaptation. We evaluate LGSE on three NLP tasks: Question Answering, Named Entity Recognition, and Text Classification, in two morphologically rich, low-resource languages: Amharic and Tigrinya, where morphological segmentation resources are available. Experimental results show that LGSE consistently outperforms baseline methods across all tasks, demonstrating the effectiveness of morphologically grounded embedding initialization for improving representation quality in underrepresented languages. Project resources are available in the GitHub link.
Abstract (translated)
将预训练语言模型适配到低资源、形态丰富的语言仍是一项重大挑战。现有词汇扩展方法通常依赖任意分割的子词单元，导致词汇表示碎片化并丢失关键的形态学信息。为应对这一局限，我们提出了词汇锚定子词嵌入初始化（LGSE）框架，该框架引入基于形态学的分割来初始化新词元的嵌入。LGSE不使用随机向量或任意子词，而是将单词分解为构成语素，并通过平均预训练的子词或基于FastText的语素表示来构建语义连贯的嵌入。当词元无法分割为有意义的语素时，则使用字符n-gram表示来捕获结构信息以构建其嵌入。在语言自适应预训练过程中，我们应用一个正则化项来惩罚新引入嵌入与初始化值之间的较大偏差，从而在保持与原始预训练嵌入空间对齐的同时实现目标语言适配。为隔离初始化效果，我们保留原始预训练模型的词汇表和分词器，并在适配过程中仅更新新嵌入。我们在两种形态丰富且资源稀缺的语言（阿姆哈拉语和提格里尼亚语，其形态分割资源可用）上，于三个NLP任务（问答、命名实体识别和文本分类）中评估了LGSE。实验结果表明，LGSE在所有任务上均持续优于基线方法，证明了基于形态学的嵌入初始化对提升低资源语言表示质量的有效性。项目资源可在GitHub链接获取。
URL
https://arxiv.org/abs/2603.22629
