Abstract
In natural language processing, multilingual models like mBERT and XLM-RoBERTa promise broad coverage but often struggle with languages that share a script yet differ in orthographic norms and cultural context. This issue is especially notable in Arabic-script languages such as Kurdish Sorani, Arabic, Persian, and Urdu. We introduce the Arabic Script RoBERTa (AS-RoBERTa) family: four RoBERTa-based models, each pre-trained on a large corpus tailored to its specific language. By focusing pre-training on language-specific script features and statistics, our models capture patterns overlooked by general-purpose models. When fine-tuned on classification tasks, AS-RoBERTa variants outperform mBERT and XLM-RoBERTa by 2 to 5 percentage points. An ablation study confirms that script-focused pre-training is central to these gains. Error analysis using confusion matrices shows how shared script traits and domain-specific content affect performance. Our results highlight the value of script-aware specialization for languages using the Arabic script and support further work on pre-training strategies rooted in script and language specificity.
Abstract (translated)
在自然语言处理领域，像mBERT和XLM-RoBERTa这样的多语种模型承诺广泛覆盖但往往难以应对那些使用相同书写系统却在正字法规范和文化背景上存在差异的语言。这一问题特别突出地表现在阿拉伯字母使用的语言中，如库尔德索拉尼语、阿拉伯语、波斯语和乌尔都语。为此，我们推出了阿拉伯文字RoBERTa（AS-RoBERTa）家族：四个基于RoBERTa的模型，每个模型都是在特定语言的大规模语料库上进行预训练的结果。通过将预训练聚焦于语言特有的书写系统特征和统计信息，我们的模型能够捕捉到通用模型所忽略的语言模式。当针对分类任务进行微调时，AS-RoBERTa变体比mBERT和XLM-RoBERTa的性能高出2至5个百分点。一项消融研究证实了以书写系统为中心的预训练对于这些改进至关重要。利用混淆矩阵进行错误分析揭示了共享书写字母特征及特定领域内容如何影响模型表现。我们的研究成果强调了针对使用阿拉伯字母的语言，具备书写意识的专业化的重要性，并支持进一步探索根植于书写系统和语言特性的预训练策略。
URL
https://arxiv.org/abs/2507.18762
