Abstract
Large Language models (LLMs) have demonstrated impressive performance on a wide range of tasks, including in multimodal settings such as speech. However, their evaluation is often limited to English and a few high-resource languages. For low-resource languages, there is no standardized evaluation benchmark. In this paper, we address this gap by introducing mSTEB, a new benchmark to evaluate the performance of LLMs on a wide range of tasks covering language identification, text classification, question answering, and translation tasks on both speech and text modalities. We evaluated the performance of leading LLMs such as Gemini 2.0 Flash and GPT-4o (Audio) and state-of-the-art open models such as Qwen 2 Audio and Gemma 3 27B. Our evaluation shows a wide gap in performance between high-resource and low-resource languages, especially for languages spoken in Africa and Americas/Oceania. Our findings show that more investment is needed to address their under-representation in LLMs coverage.
Abstract (translated)
大型语言模型（LLMs）在包括语音在内的多种模态任务中展现出了卓越的性能。然而，这些模型的评估通常仅限于英语及其他几种资源丰富的语言。对于资源匮乏的语言，缺乏标准化的评估基准。为解决这一问题，本文引入了mSTEB这一新的评估基准，用以衡量LLMs在语言识别、文本分类、问答和翻译任务上的表现，覆盖语音和文本模态的广泛任务。我们对领先的大规模模型（如Gemini 2.0 Flash 和 GPT-4o (Audio)）以及最先进的开源模型（如Qwen 2 Audio 和 Gemma 3 27B）进行了评估。我们的研究结果表明，在资源丰富与匮乏的语言之间，尤其是在非洲和美洲/大洋洲使用的语言上，LLMs的表现存在显著差距。这一发现表明，需要更多的投资来解决这些语言在LLMs覆盖中的不足问题。
URL
https://arxiv.org/abs/2506.08400
