Abstract
While gender bias in modern Neural Machine Translation (NMT) systems has received much attention, traditional evaluation metrics do not to fully capture the extent to which these systems integrate contextual gender cues. We propose a novel evaluation metric called Minimal Pair Accuracy (MPA), which measures the reliance of models on gender cues for gender disambiguation. MPA is designed to go beyond surface-level gender accuracy metrics by focusing on whether models adapt to gender cues in minimal pairs -- sentence pairs that differ solely in the gendered pronoun, namely the explicit indicator of the target's entity gender in the source language (EN). We evaluate a number of NMT models on the English-Italian (EN--IT) language pair using this metric, we show that they ignore available gender cues in most cases in favor of (statistical) stereotypical gender interpretation. We further show that in anti-stereotypical cases, these models tend to more consistently take masculine gender cues into account while ignoring the feminine cues. Furthermore, we analyze the attention head weights in the encoder component and show that while all models encode gender information to some extent, masculine cues elicit a more diffused response compared to the more concentrated and specialized responses to feminine gender cues.
Abstract (translated)
尽管现代神经机器翻译（NMT）系统中的性别偏见已受到广泛关注，但传统的评估指标并不能完全捕捉这些系统在整合上下文性别线索方面的程度。为此，我们提出了一种新的评估指标——最小成对准确度（MPA），该指标衡量模型依赖于性别线索进行性别消歧的程度。MPA旨在超越表面的性别准确性指标，通过关注模型如何根据最小对中的性别代词调整其行为来实现这一点——即源语言中目标实体性别的明确指示器。 我们使用这一新指标评估了多种NMT模型在英语-意大利语（EN--IT）语言对上的性能。结果表明，在大多数情况下，这些模型倾向于忽略可用的性别线索而依赖于统计意义上的刻板印象进行性别解读。此外，我们在反向刻板印象的情况下还发现，这些模型更一致地考虑男性性别线索并忽视女性性别线索。 进一步分析编码器组件中的注意力头权重后我们发现，虽然所有模型都在某种程度上编码了性别信息，但男性提示会引发更为分散的响应，而女性性别的提示则引发了更加集中和专业的反应。
URL
https://arxiv.org/abs/2505.08546
