Abstract
Large language models (LLMs) are advancing at an unprecedented pace globally, with regions increasingly adopting these models for applications in their primary language. Evaluation of these models in diverse linguistic environments, especially in low-resource languages, has become a major challenge for academia and industry. Existing evaluation frameworks are disproportionately focused on English and a handful of high-resource languages, thereby overlooking the realistic performance of LLMs in multilingual and lower-resource scenarios. To address this gap, we introduce GlotEval, a lightweight framework designed for massively multilingual evaluation. Supporting seven key tasks (machine translation, text classification, summarization, open-ended generation, reading comprehension, sequence labeling, and intrinsic evaluation), spanning over dozens to hundreds of languages, GlotEval highlights consistent multilingual benchmarking, language-specific prompt templates, and non-English-centric machine translation. This enables a precise diagnosis of model strengths and weaknesses in diverse linguistic contexts. A multilingual translation case study demonstrates GlotEval's applicability for multilingual and language-specific evaluations.
Abstract (translated)
大型语言模型（LLMs）在全球范围内以前所未有的速度发展，各地区越来越多地采用这些模型来支持其主要语言的应用。在多语种环境中特别是低资源语言中评估这些模型已成为学术界和工业界的巨大挑战。现有的评估框架过于侧重于英语和其他几种高资源语言，从而忽略了大规模多语言及低资源场景下大型语言模型的真实性能表现。为解决这一问题，我们引入了GlotEval，这是一个轻量级的框架，旨在进行大规模多语种评估。该框架支持七项关键任务（机器翻译、文本分类、摘要生成、开放性生成、阅读理解、序列标注和内在评测），覆盖数十到数百种语言，强调一致性的跨语言基准测试、语言特定的提示模板以及非英语中心化的机器翻译。这使得能够精确诊断模型在不同语言环境中的优势与劣势。 一个多语种翻译案例研究展示了GlotEval在多语言及语言特有评估方面的适用性。
URL
https://arxiv.org/abs/2504.04155
