Abstract
The rapid advancement of Large Language Models (LLMs), particularly those trained on multilingual corpora, has intensified the need for a deeper understanding of their performance across a diverse range of languages and model sizes. Our research addresses this critical need by studying the performance and scaling behavior of multilingual LLMs in text classification and machine translation tasks across 204 languages. We systematically examine both seen and unseen languages across three model families of varying sizes in zero-shot and few-shot settings. Our findings show significant differences in scaling behavior between zero-shot and two-shot scenarios, with striking disparities in performance between seen and unseen languages. Model scale has little effect on zero-shot performance, which remains mostly flat. However, in two-shot settings, larger models show clear linear improvements in multilingual text classification. For translation tasks, however, only the instruction-tuned model showed clear benefits from scaling. Our analysis also suggests that overall resource levels, not just the proportions of pretraining languages, are better predictors of model performance, shedding light on what drives multilingual LLM effectiveness.
Abstract (translated)
大型语言模型（LLMs）的迅速发展，尤其是那些基于多语种语料库训练的模型，增强了对其在不同语言和规模范围内性能进行深入理解的需求。我们的研究通过探讨204种语言中多语种LLMs在文本分类和机器翻译任务中的表现及扩展行为，来解决这一关键需求。我们系统地考察了三种大小不同的模型家族在零样本和少量样本设置下已见与未见过的语言的性能。 我们的发现表明，在零样本和两样本情景之间存在显著的扩展行为差异，并且已见语言与未见过语言之间的表现差距明显。对于零样本情境，模型规模对性能影响不大，大多数情况下保持不变；然而在两样本设定中，较大的模型在多语种文本分类方面表现出明显的线性改进。但对于翻译任务而言，只有经过指令调优的模型从扩展中明确受益。 我们的分析还表明，整体资源水平（而不仅仅是预训练语言的比例）是更好的性能预测指标，这为理解驱动多语言LLMs有效性的因素提供了洞见。
URL
https://arxiv.org/abs/2501.05629
