Abstract
Attention maps in neural models for NLP are appealing to explain the decision made by a model, hopefully emphasizing words that justify the decision. While many empirical studies hint that attention maps can provide such justification from the analysis of sound examples, only a few assess the plausibility of explanations based on attention maps, i.e., the usefulness of attention maps for humans to understand the decision. These studies furthermore focus on text classification. In this paper, we report on a preliminary assessment of attention maps in a sentence comparison task, namely natural language inference. We compare the cross-attention weights between two RNN encoders with human-based and heuristic-based annotations on the eSNLI corpus. We show that the heuristic reasonably correlates with human annotations and can thus facilitate evaluation of plausible explanations in sentence comparison tasks. Raw attention weights however remain only loosely related to a plausible explanation.
Abstract (translated)
神经模型中的注意力图（Attention maps）在解释模型决策方面具有吸引力，理想情况下应强调那些能为决策提供依据的词语。虽然许多实证研究表明，通过对典型示例的分析可以得出这样的结论，即注意力图能够为模型决策提供合理说明，但只有少数研究评估了基于注意力图的解释的实际有效性，也就是这些解释对于人类理解模型决策是否有用。而且这些研究主要集中在文本分类任务上。 本文报告了一项初步研究，在句子比较任务（特别是自然语言推理）中对注意力图进行了评估。我们对比了两个RNN编码器之间的交叉注意权重，并将其与eSNLI语料库中的基于人的注释和基于启发式的注释进行对照。结果表明，基于启发式的方法合理地与人类的标注相关联，因此可以在句子比较任务中促进可接受解释的评估。然而，原始注意力权重仍然仅与合理的解释存在松散的相关性。
URL
https://arxiv.org/abs/2501.13735
