Abstract
India's linguistic landscape is one of the most diverse in the world, comprising over 120 major languages and approximately 1,600 additional languages, with 22 officially recognized as scheduled languages in the Indian Constitution. Despite recent progress in multilingual neural machine translation (NMT), high-quality parallel corpora for Indian languages remain scarce, especially across varied domains. In this paper, we introduce a large-scale, high-quality annotated parallel corpus covering 11 of these languages : English, Telugu, Hindi, Punjabi, Odia, Kashmiri, Sindhi, Dogri, Kannada, Urdu, and Gujarati comprising a total of 772,000 bi-text sentence pairs. The dataset is carefully curated and systematically categorized into three key domains: Government, Health, and General, to enable domain-aware machine translation research and facilitate effective domain adaptation. To demonstrate the utility of CorIL and establish strong benchmarks for future research, we fine-tune and evaluate several state-of-the-art NMT models, including IndicTrans2, NLLB, and BhashaVerse. Our analysis reveals important performance trends and highlights the corpus's value in probing model capabilities. For instance, the results show distinct performance patterns based on language script, with massively multilingual models showing an advantage on Perso-Arabic scripts (Urdu, Sindhi) while other models excel on Indic scripts. This paper provides a detailed domain-wise performance analysis, offering insights into domain sensitivity and cross-script transfer learning. By publicly releasing CorIL, we aim to significantly improve the availability of high-quality training data for Indian languages and provide a valuable resource for the machine translation research community.
Abstract (translated)
印度的语言景观是世界上最多样化的一种，包含了超过120种主要语言和大约1600种其他语言。根据《印度宪法》，其中有22种语言被列为计划语言（scheduled languages）。尽管近期在多语神经机器翻译(NMT)方面取得了一些进展，但在各种领域中高质量的平行语料库仍然十分稀缺。在这篇论文中，我们介绍了一个大规模且高质量标注的平行语料库，涵盖了11种印度的语言：英语、泰卢固语、印地语、旁遮普语、奥迪亚语、克什米尔语、信德语、多格里语、卡纳达语、乌尔都语和古吉拉特语。这个语料库总共包含772,000对双语文本句子。 该数据集经过精心策划，并被系统地分类为三大关键领域：政府、健康以及通用，以支持具有域意识的机器翻译研究并促进有效的领域适应。为了展示CorIL（可能是指上述提到的大规模平行语料库）的实用性，并为未来的研究建立强大的基准，我们对几种最先进的NMT模型进行了微调和评估，包括IndicTrans2、NLLB和BhashaVerse。我们的分析揭示了重要的性能趋势，并突出了该语料库在探索模型能力方面的价值。例如，结果表明，基于书写系统的不同语言模式具有明显的性能差异：大规模多语种模型在使用波斯-阿拉伯书写的语言（如乌尔都语、信德语）上表现出优势，而其他模型则在印度书写系统上的表现更为出色。 本文还提供了详细的领域内性能分析，为理解领域的敏感性和跨书写系统的迁移学习提供见解。通过公开发布CorIL，我们旨在显著提高印度语言高质量训练数据的可用性，并为此领域的机器翻译研究社区提供宝贵的资源。
URL
https://arxiv.org/abs/2509.19941
