Abstract
We introduce MT-LENS, a framework designed to evaluate Machine Translation (MT) systems across a variety of tasks, including translation quality, gender bias detection, added toxicity, and robustness to misspellings. While several toolkits have become very popular for benchmarking the capabilities of Large Language Models (LLMs), existing evaluation tools often lack the ability to thoroughly assess the diverse aspects of MT performance. MT-LENS addresses these limitations by extending the capabilities of LM-eval-harness for MT, supporting state-of-the-art datasets and a wide range of evaluation metrics. It also offers a user-friendly platform to compare systems and analyze translations with interactive visualizations. MT-LENS aims to broaden access to evaluation strategies that go beyond traditional translation quality evaluation, enabling researchers and engineers to better understand the performance of a NMT model and also easily measure system's biases.
Abstract (translated)
我们介绍了一种名为MT-LENS的框架，该框架旨在评估机器翻译（MT）系统在各种任务中的表现，包括翻译质量、性别偏见检测、毒性增加以及对拼写错误的鲁棒性。虽然已有多种工具包广泛用于基准测试大型语言模型（LLMs）的能力，但现有的评估工具往往缺乏全面评估MT性能多样性的能力。MT-LENS通过扩展LM-eval-harness以支持机器翻译，解决了这些局限性，并且兼容最先进的数据集和广泛的评估指标。它还提供了一个用户友好的平台来比较系统并使用交互式可视化分析翻译结果。MT-LENS旨在拓宽对传统翻译质量评价之外的评估策略的访问权限，使研究人员和工程师能够更好地理解NMT模型的表现，并轻松测量系统的偏见。
URL
https://arxiv.org/abs/2412.11615
