Abstract
Multilingual pretraining typically lacks explicit alignment signals, leading to suboptimal cross-lingual alignment in the representation space. In this work, we show that training standard pretrained models for cross-lingual alignment with a multi-way parallel corpus in a diverse pool of languages can substantially improve multilingual and cross-lingual representations for NLU tasks. We construct a multi-way parallel dataset using translations of English text from an off-the-shelf NMT model for a pool of six target languages and achieve strong cross-lingual alignment through contrastive learning. This leads to substantial performance gains across both seen and unseen languages for multiple tasks from the MTEB benchmark evaluated for XLM-Roberta and multilingual BERT base models. Using a multi-way parallel corpus for contrastive training yields substantial gains on bitext mining (21.3%), semantic similarity (5.3%), and classification (28.4%) compared to English-centric (En-X) bilingually parallel data, where X is sampled from a pool of multiple target languages. Furthermore, finetuning mE5 model on a small dataset with multi-way parallelism significantly improves bitext mining compared to one without, underscoring the importance of multi-way cross-lingual supervision even for models already pretrained for high-quality sentence embeddings.
Abstract (translated)
多语言预训练通常缺乏显式的对齐信号，导致跨语言表示空间中的对齐效果不佳。在本工作中，我们展示了使用一个多向平行语料库（包含多种语言）来训练标准的预训练模型可以显著提升用于自然语言理解任务的多语言和跨语言表示。我们构建了一个多向平行数据集，通过现成的神经机器翻译模型将英语文本翻译为六种目标语言，并采用对比学习实现了强大的跨语言对齐效果。这使得XLM-Roberta和多语言BERT基础模型在MTEB基准测试中的多个任务上，无论是已见还是未见过的语言都取得了显著的表现提升。 与以英语为中心的（En-X）双语平行数据相比，使用多向平行语料库进行对比训练在双语文本挖掘、语义相似度和分类方面分别获得了21.3%、5.3%和28.4%的大幅提升。其中X是从多个目标语言中抽取的一种语言。此外，在一个小规模数据集上对mE5模型进行微调，使用多向平行语料库相比于不使用的方案显著提高了双语文本挖掘的效果，强调了即使对于已经预训练出高质量句子嵌入的模型来说，多向跨语言监督的重要性。
URL
https://arxiv.org/abs/2602.21543
