Abstract
Recent advances in word embeddings and language models use large-scale, unlabelled data and self-supervised learning to boost NLP performance. Multilingual models, often trained on web-sourced data like Wikipedia, face challenges: few low-resource languages are included, their data is often noisy, and lack of labeled datasets makes it hard to evaluate performance outside high-resource languages like English. In this dissertation, we focus on languages spoken in Sub-Saharan Africa where all the indigenous languages in this region can be regarded as low-resourced in terms of the availability of labelled data for NLP tasks and unlabelled data found on the web. We analyse the noise in the publicly available corpora, and curate a high-quality corpus, demonstrating that the quality of semantic representations learned in word embeddings does not only depend on the amount of data but on the quality of pre-training data. We demonstrate empirically the limitations of word embeddings, and the opportunities the multilingual pre-trained language model (PLM) offers especially for languages unseen during pre-training and low-resource scenarios. We further study how to adapt and specialize multilingual PLMs to unseen African languages using a small amount of monolingual texts. To address the under-representation of the African languages in NLP research, we developed large scale human-annotated labelled datasets for 21 African languages in two impactful NLP tasks: named entity recognition and machine translation. We conduct an extensive empirical evaluation using state-of-the-art methods across supervised, weakly-supervised, and transfer learning settings.
Abstract (translated)
近期在词嵌入和语言模型方面的进展利用大规模无标注数据及自监督学习来提升自然语言处理（NLP）的性能。多语种模型通常基于如维基百科等网络来源的数据进行训练，但面临着几个挑战：少数低资源语言未被包括、其使用的数据往往质量不高且含有噪声，缺乏标记的数据集使得评估除英语这类高资源语言之外的语言表现变得困难。在这篇博士论文中，我们专注于撒哈拉以南非洲地区所使用的所有土著语言，在NLP任务的标注数据和网络上可用的无标注数据方面均被视为低资源语言。我们分析了公开语料库中的噪声，并整理出高质量的语料库，表明词嵌入学习到的语义表示不仅依赖于数据量还取决于预训练数据的质量。通过实证展示了词嵌入的局限性以及多语种预训练语言模型（PLM）在未见过的语言及低资源场景中提供的机会。我们进一步研究了如何使用少量单语文本来适应和专门化多语种PLM以应对非洲未知语言。为了弥补非洲语言在NLP研究中的代表性不足，我们为21个非洲语言开发了一整套大规模的人工注释标注数据集，在两个具有影响力的任务中：命名实体识别与机器翻译任务中使用这些数据集进行测试。我们采用最新的方法进行了广泛的实证评估，涵盖了监督学习、弱监督学习及迁移学习设置。
URL
https://arxiv.org/abs/2507.00297
