Abstract
Advances in transformer-based language models have highlighted the benefits of language-specific pre-training on high-quality corpora. In this context, German NLP stands to gain from updated architectures and modern datasets tailored to the linguistic characteristics of the German language. GeistBERT seeks to improve German language processing by incrementally training on a diverse corpus and optimizing model performance across various NLP tasks. It was pre-trained using fairseq with standard hyperparameters, initialized from GottBERT weights, and trained on a large-scale German corpus using Whole Word Masking (WWM). Based on the pre-trained model, we derived extended-input variants using Nyströmformer and Longformer architectures with support for sequences up to 8k tokens. While these long-context models were not evaluated on dedicated long-context benchmarks, they are included in our release. We assessed all models on NER (CoNLL 2003, GermEval 2014) and text classification (GermEval 2018 fine/coarse, 10kGNAD) using $F_1$ score and accuracy. The GeistBERT models achieved strong performance, leading all tasks among the base models and setting a new state-of-the-art (SOTA). Notably, the base models outperformed larger models in several tasks. To support the German NLP research community, we are releasing GeistBERT under the MIT license.
Abstract (translated)
基于变压器的语言模型的进展强调了在高质量语料库上进行语言特定预训练的好处。在此背景下，德语文本处理可以从更新的架构和专为德语语言特点设计的现代数据集中获益。GeistBERT旨在通过在多样化语料库上逐步训练并优化各种NLP任务中的模型性能来改进德文语言处理。它使用fairseq进行预训练，并采用标准超参数，从GottBERT权重初始化，然后使用Whole Word Masking（WWM）技术在一个大规模的德语文本语料库上进行训练。 基于这个预训练模型，我们利用Nyströmformer和Longformer架构生成了支持最长达8000个token序列的扩展输入变体。尽管这些长上下文模型并未在专门针对长上下文基准测试中评估，但它们已包含在我们的发布版本中。 我们在命名实体识别（CoNLL 2003, GermEval 2014）和文本分类（GermEval 2018细/粗粒度，10kGNAD）任务上对所有模型进行了测试，并使用$F_1$分数和准确率作为评价标准。GeistBERT模型在所有基础模型中均表现出色，在多个任务中设立了新的最佳状态（SOTA）。值得注意的是，在某些任务中，基础模型的表现优于更大的模型。 为了支持德语文本处理研究社区的发展，我们将在MIT许可证下发布GeistBERT。
URL
https://arxiv.org/abs/2506.11903
