Abstract
The conversion of content from one language to another utilizing a computer system is known as Machine Translation (MT). Various techniques have come up to ensure effective translations that retain the contextual and lexical interpretation of the source language. End-to-end Neural Machine Translation (NMT) is a popular technique and it is now widely used in real-world MT systems. Massive amounts of parallel datasets (sentences in one language alongside translations in another) are required for MT systems. These datasets are crucial for an MT system to learn linguistic structures and patterns of both languages during the training phase. One such dataset is Samanantar, the largest publicly accessible parallel dataset for Indian languages (ILs). Since the corpus has been gathered from various sources, it contains many incorrect translations. Hence, the MT systems built using this dataset cannot perform to their usual potential. In this paper, we propose an algorithm to remove mistranslations from the training corpus and evaluate its performance and efficiency. Two Indic languages (ILs), namely, Hindi (HIN) and Odia (ODI) are chosen for the experiment. A baseline NMT system is built for these two ILs, and the effect of different dataset sizes is also investigated. The quality of the translations in the experiment is evaluated using standard metrics such as BLEU, METEOR, and RIBES. From the results, it is observed that removing the incorrect translation from the dataset makes the translation quality better. It is also noticed that, despite the fact that the ILs-English and English-ILs systems are trained using the same corpus, ILs-English works more effectively across all the evaluation metrics.
Abstract (translated)
将一种语言的内容转换为另一种语言的内容利用计算机系统进行翻译被称为机器翻译（MT）。为了确保有效的翻译并保留源语言的上下文和词汇解释，已经提出了各种技术。端到端神经机器翻译（NMT）是一种流行的技术，现在在现实世界的MT系统中得到了广泛应用。大量的并行数据集（一种语言的句子及另一种语言的翻译）对于MT系统来说至关重要。 Samanantar 是一个公开可用的针对印度语言的大型并行数据集（ILs）。由于从各种来源收集，因此其中包含许多错误翻译。因此，使用这个数据集构建的MT系统无法发挥其通常的功能。在本文中，我们提出了一个算法来从训练语料库中消除错误翻译，并评估其性能和效率。我们选择了两种IL，即印地语（HIN）和奥迪亚语（ODI）进行实验。为这两个IL构建了 baseline NMT系统，并研究了不同数据集大小对系统性能的影响。实验中使用的翻译质量评估标准包括BLEU、METEOR和RIBES等。从结果中观察到，从数据集中移除错误翻译可以提高翻译质量。此外，还发现IL-英语系统在所有评估指标上都比ENGL-IL系统更有效。
URL
https://arxiv.org/abs/2401.06398
