Abstract
This paper analyses how traditional baseline metrics, such as BLEU and TER, and neural-based methods, such as BERTScore and COMET, score several NMT models performance on chat translation and how these metrics perform when compared to human-annotated scores. The results show that for ranking NMT models in chat translations, all metrics seem consistent in deciding which model outperforms the others. This implies that traditional baseline metrics, which are faster and simpler to use, can still be helpful. On the other hand, when it comes to better correlation with human judgment, neural-based metrics outperform traditional metrics, with COMET achieving the highest correlation with the human-annotated score on a chat translation. However, we show that even the best metric struggles when scoring English translations from sentences with anaphoric zero-pronoun in Japanese.
Abstract (translated)
本文分析了传统基准指标（如BLEU和TER）以及基于神经网络的方法（如BERTScore和COMET）如何评估几种NMT模型在聊天翻译中的性能，以及这些指标与人工标注得分相比的表现如何。结果显示，在对聊天翻译中NMT模型进行排名时，所有指标在决定哪个模型表现更优方面似乎是一致的。这表明传统的基准指标由于使用更快、更简单，仍然可以是有用的。另一方面，当涉及到更好地与人类判断相关联时，基于神经网络的指标优于传统指标，在聊天翻译中COMET与人工标注得分的相关性最高。然而，我们还发现，即使是最优的指标在对包含日语中的指示代词零代词的句子进行英语翻译评分时也显得力不从心。
URL
https://arxiv.org/abs/2412.18190
