Abstract
Idiomatic translation remains a significant challenge in machine translation, especially for low resource languages such as Urdu, and has received limited prior attention. To advance research in this area, we introduce the first evaluation datasets for Urdu to English idiomatic translation, covering both Native Urdu and Roman Urdu scripts and annotated with gold-standard English equivalents. We evaluate multiple open-source Large Language Models (LLMs) and Neural Machine Translation (NMT) systems on this task, focusing on their ability to preserve idiomatic and cultural meaning. Automatic metrics including BLEU, BERTScore, COMET, and XCOMET are used to assess translation quality. Our findings indicate that prompt engineering enhances idiomatic translation compared to direct translation, though performance differences among prompt types are relatively minor. Moreover, cross script comparisons reveal that text representation substantially affects translation quality, with Native Urdu inputs producing more accurate idiomatic translations than Roman Urdu.
Abstract (translated)
习语翻译在机器翻译中仍是一项重大挑战，尤其是在像乌尔都语这样的低资源语言领域，这一问题尚未得到充分关注。为了推动该领域的研究进展，我们首次推出了用于乌尔都语到英语习语翻译的评估数据集，涵盖了本土乌尔都文和罗马化乌尔都文两种书写系统，并附有黄金标准的英文对照译文。我们在该项任务中评估了多个开源大型语言模型（LLMs）和神经机器翻译（NMT）系统的性能，重点关注它们在保留习语及文化含义方面的能力。我们使用包括BLEU、BERTScore、COMET和XCOMET在内的自动评价指标来衡量翻译质量。 研究发现表明，与直接翻译相比，提示工程能显著提升习语翻译的效果，尽管不同类型的提示之间存在细微的性能差异。此外，在跨书写系统对比中发现，文本表示对翻译质量有着重要影响：基于本土乌尔都文输入的翻译比罗马化乌尔都文更准确地保留了习语的意义和文化内涵。
URL
https://arxiv.org/abs/2510.17460
