Abstract
Recent advancements in natural language processing have led to the proliferation of large language models (LLMs). These models have been shown to yield good performance, using in-context learning, even on unseen tasks and languages. Additionally, they have been widely adopted as language-model-as-a-service commercial APIs like GPT-4 API. However, their performance on African languages is largely unknown. We present an analysis of three popular large language models (mT0, LLaMa 2, and GPT-4) on five tasks (news topic classification, sentiment classification, machine translation, question answering, and named entity recognition) across 30 African languages, spanning different language families and geographical regions. Our results suggest that all LLMs produce below-par performance on African languages, and there is a large gap in performance compared to high-resource languages like English most tasks. We find that GPT-4 has an average or impressive performance on classification tasks but very poor results on generative tasks like machine translation. Surprisingly, we find that mT0 had the best overall on cross-lingual QA, better than the state-of-the-art supervised model (i.e. fine-tuned mT5) and GPT-4 on African languages. Overall, LLaMa 2 records the worst performance due to its limited multilingual capabilities and English-centric pre-training corpus. In general, our findings present a call-to-action to ensure African languages are well represented in large language models, given their growing popularity.
Abstract (translated)
近年来自然语言处理领域的进步导致了大型语言模型的（LLMs）的繁荣。这些模型已经在可见的任务和语言上表现良好，即使是在未见过的任务和语言上。此外，它们已经被广泛应用于诸如GPT-4 API这样的语言模型服务中。然而，它们在非洲语言上的表现仍然是未知的。我们对三种流行的LLM（mT0、LLaMa 2和GPT-4）在30个非洲语言上的五个任务（新闻主题分类、情感分类、机器翻译、问答和命名实体识别）进行了分析。我们的结果表明，所有LLM在非洲语言上的表现都低于预期，与英语等高资源语言相比，差距很大。我们发现，GPT-4在分类任务上具有平均或出色的性能，但在生成任务（如机器翻译）上表现非常差。令人惊讶的是，我们发现mT0在跨语言QA上表现最佳，优于当前最先进的监督模型（即微调的mT5）和GPT-4在非洲语言上的表现。总的来说，LLaMa 2由于其有限的多语言能力以及英语中心化的预训练语料库而记录了最差的表现。总的来说，我们的研究结果发出一个呼吁，确保非洲语言在大型语言模型中得到充分代表，鉴于它们日益增长的重要性。
URL
https://arxiv.org/abs/2311.07978
