Abstract
Neural Machine Translation (NMT) models are typically trained on datasets with limited exposure to Scientific, Technical and Educational domains. Translation models thus, in general, struggle with tasks that involve scientific understanding or technical jargon. Their performance is found to be even worse for low-resource Indian languages. Finding a translation dataset that tends to these domains in particular, poses a difficult challenge. In this paper, we address this by creating a multilingual parallel corpus containing more than 2.8 million rows of English-to-Indic and Indic-to-Indic high-quality translation pairs across 8 Indian languages. We achieve this by bitext mining human-translated transcriptions of NPTEL video lectures. We also finetune and evaluate NMT models using this corpus and surpass all other publicly available models at in-domain tasks. We also demonstrate the potential for generalizing to out-of-domain translation tasks by improving the baseline by over 2 BLEU on average for these Indian languages on the Flores+ benchmark. We are pleased to release our model and dataset via this link: this https URL.
Abstract (translated)
神经机器翻译（NMT）模型通常是在对科学、技术和教育领域暴露有限的数据集上进行训练的。因此，这些模型在涉及科学理解或技术术语的任务中往往表现不佳。对于低资源印度语言来说，其性能甚至更差。找到一个特别倾向于这些领域的翻译数据集是一个艰巨的挑战。在这篇论文中，我们通过创建一个多语种平行语料库来解决这个问题，该语料库包含超过280万行英-印和印-印高质量翻译对，覆盖了8种印度语言。我们通过挖掘NPTEL视频讲座的人工翻译字幕实现了这一目标。此外，我们还使用这个语料库微调并评估NMT模型，并在领域内任务上超越了所有其他公开可用的模型。我们还展示了该模型在外域翻译任务中的泛化能力，在Flores+基准测试中，这些印度语言的基线平均提高了超过2个BLEU分数。我们很高兴通过以下链接发布我们的模型和数据集：此 https URL。
URL
https://arxiv.org/abs/2412.09025
