Abstract
Large language models often underperform in many European languages due to the dominance of English and a few high-resource languages in training data. This paper presents TildeOpen LLM, a 30-billion-parameter open-weight foundational model trained for 34 European languages to promote linguistic equity and improve performance for low-resource languages. To address the data imbalance, we combine dataset upsampling with a curriculum-based training schedule that alternates between uniform and natural language distributions. The resulting model performs favorably compared to other multilingual LLMs despite being trained with significantly fewer computing resources. Evaluation across multiple multilingual benchmarks shows that TildeOpen surpasses existing open-weight models in text generation and comprehension, particularly for Baltic, Finno-Ugric, and Slavic languages. Human evaluations confirm an up to tenfold reduction in linguistic errors relative to leading baselines. The model and associated resources are fully open-weight and publicly available at this http URL. These outcomes demonstrate that careful data curation and balanced training strategies can substantially enhance multilingual model quality without increasing model size or training volume.
Abstract (translated)
大型语言模型在许多欧洲语言中的表现通常不如人意，这是因为训练数据中英语和其他几种高资源语言占主导地位。本文介绍了TildeOpen LLM，这是一个为34种欧洲语言训练的300亿参数的基础开放权重模型，旨在促进语言平等并提高低资源语言的表现。为了应对数据不平衡问题，我们结合了数据集上采样与基于课程的学习时间表，该时间表在均匀分布和自然语言分布之间交替进行。 尽管使用计算资源显著较少，但最终生成的模型与其他多语种LLM相比表现良好。多项跨多种语言基准测试表明，TildeOpen 在文本生成和理解方面超越了现有的开放权重模型，特别是在波罗的语系、乌拉尔-芬兰语系和斯拉夫语系语言中的性能尤为突出。人工评估确认，在与领先基线相比时，语言错误减少了多达十倍。 该模型及相关资源完全为开放权重，并且公开可用（可在此网址访问：[this http URL]）。这些成果表明，通过仔细的数据整理和平衡的训练策略，可以在不增加模型大小或训练量的情况下显著提高多语种模型的质量。
URL
https://arxiv.org/abs/2603.08182
