Abstract
While BERT is an effective method for learning monolingual sentence embeddings for semantic similarity and embedding based transfer learning BERT based cross-lingual sentence embeddings have yet to be explored. We systematically investigate methods for learning multilingual sentence embeddings by combining the best methods for learning monolingual and cross-lingual representations including: masked language modeling (MLM), translation language modeling (TLM), dual encoder translation ranking, and additive margin softmax. We show that introducing a pre-trained multilingual language model dramatically reduces the amount of parallel training data required to achieve good performance by 80%. Composing the best of these methods produces a model that achieves 83.7% bi-text retrieval accuracy over 112 languages on Tatoeba, well above the 65.5 achieved by LASER, while still performing competitively on monolingual transfer learning benchmarks. Parallel data mined from CommonCrawl using our best model is shown to train competitive NMT models for en-zh and en-de. We publicly release our best multilingual sentence embedding model for 109+ languages at this https URL.
Abstract (translated)
虽然BERT在学习单语句子嵌入以进行语义相似度和基于嵌入的迁移学习方面非常有效，但基于BERT的跨语言句子嵌入尚未得到充分研究。我们系统地调查了结合最佳单语文本表示与跨语言文本表示方法来学习多语言句子嵌入的方法，包括掩码语言模型（MLM）、翻译语言模型（TLM）、双编码器翻译排名以及加性余弦softmax。我们表明，引入预训练的多语种语言模型可以显著减少实现良好性能所需的平行训练数据量达80%。将这些方法中的最佳组合产生了一个在Tatoeba上涵盖112种语言且双文本检索准确率达到83.7%的模型，这一成绩远超LASER所达到的65.5%，同时该模型仍能在单语迁移学习基准测试中保持竞争力。使用我们的最佳模型从CommonCrawl中挖掘出的平行数据训练了竞争性的英汉和英德NMT（神经机器翻译）模型。我们公开发布了适用于109+种语言的最佳多语言句子嵌入模型，可在此URL下载：[https://github.com/your-repo-url](https://github.com/your-repo-url)。 请注意，在提供的链接处应填入实际的GitHub仓库地址或其他发布平台地址以供访问。
URL
https://arxiv.org/abs/2510.17504
