Abstract
Current machine translation models provide us with high-quality outputs in most scenarios. However, they still face some specific problems, such as detecting which entities should not be changed during translation. In this paper, we explore the abilities of popular NMT models, including models from the OPUS project, Google Translate, MADLAD, and EuroLLM, to preserve entities such as URL addresses, IBAN numbers, or emails when producing translations between four languages: English, German, Polish, and Ukrainian. We investigate the quality of popular NMT models in terms of accuracy, discuss errors made by the models, and examine the reasons for errors. Our analysis highlights specific categories, such as emojis, that pose significant challenges for many models considered. In addition to the analysis, we propose a new multilingual synthetic dataset of 36,000 sentences that can help assess the quality of entity transfer across nine categories and four aforementioned languages.
Abstract (translated)
当前的机器翻译模型在大多数场景中提供了高质量的输出。然而，它们仍然面临一些特定问题，例如检测哪些实体在翻译过程中不应被更改。在这篇论文中，我们研究了流行神经机器翻译（NMT）模型的能力，包括来自OPUS项目、Google Translate、MADLAD和EuroLLM的模型，在四种语言之间进行翻译时保留实体（如网址地址、IBAN号码或电子邮件）的效果。这四种语言分别是英语、德语、波兰语和乌克兰语。我们分析了这些流行NMT模型在准确性方面的质量，讨论了它们所犯的错误，并探讨了导致错误的原因。我们的分析突出了诸如表情符号等特定类别，这对许多考虑中的模型构成了重大挑战。除了这项分析外，我们还提出了一套新的多语言合成数据集，包含36,000个句子，可以用来评估九种类型实体在上述四种语言之间的传输质量。
URL
https://arxiv.org/abs/2505.06010
