Abstract
We introduce MASSES, a simple evaluation metric for the task of Visual Question Answering (VQA). In its standard form, the VQA task is operationalized as follows: Given an image and an open-ended question in natural language, systems are required to provide a suitable answer. Currently, model performance is evaluated by means of a somehow simplistic metric: If the predicted answer is chosen by at least 3 human annotators out of 10, then it is 100% correct. Though intuitively valuable, this metric has some important limitations. First, it ignores whether the predicted answer is the one selected by the Majority (MA) of annotators. Second, it does not account for the quantitative Subjectivity (S) of the answers in the sample (and dataset). Third, information about the Semantic Similarity (SES) of the responses is completely neglected. Based on such limitations, we propose a multi-component metric that accounts for all these issues. We show that our metric is effective in providing a more fine-grained evaluation both on the quantitative and qualitative level.
Abstract (translated)
我们介绍了MASSES，一个用于视觉问答（VQA）任务的简单评估指标。在其标准形式中，VQA任务的操作如下：给定图像和自然语言的开放式问题，系统需要提供合适的答案。目前，通过某种简单的度量来评估模型性能：如果预测的答案是由10个中的至少3个人类注释器选择的，那么它是100％正确的。虽然直观有价值，但该指标有一些重要的局限性。首先，它忽略了预测答案是否是由注释者的多数（MA）选择的答案。其次，它没有考虑样本（和数据集）中答案的定量主观性（S）。第三，完全忽略了关于响应的语义相似性（SES）的信息。基于这些限制，我们提出了一个多组件指标来解决所有这些问题。我们表明，我们的指标可以有效地在定量和定性水平上提供更细粒度的评估。
URL
https://arxiv.org/abs/1809.04344
