Abstract
Visual question answering (VQA) models respond to open-ended natural language questions about images. While VQA is an increasingly popular area of research, it is unclear to what extent current VQA architectures learn key semantic distinctions between visually-similar images. To investigate this question, we explore a reformulation of the VQA task that challenges models to identify counterexamples: images that result in a different answer to the original question. We introduce two plug-and-play methods for evaluating existing VQA models against a supervised counterexample prediction task, VQA-CX. While our models surpass existing benchmarks on VQA-CX, we find that the multimodal representations learned by an existing state-of-the-art VQA model contribute only marginally to performance on this task. These results call into question the assumption that successful performance on the VQA benchmark is indicative of general visual-semantic reasoning abilities.
Abstract (translated)
视觉问答（VQA）模型回应关于图像的开放式自然语言问题。虽然VQA是一个越来越受欢迎的研究领域，但目前尚不清楚VQA架构在多大程度上学习视觉相似图像之间的关键语义区别。为了调查这个问题，我们探索了VQA任务的重新表述，这个任务挑战模型来识别反例：导致对原始问题的答案不同的图像。我们介绍两种即插即用方法，用于根据受监督的反例预测任务VQA-CX评估现有的VQA模型。虽然我们的模型超越了VQA-CX的现有基准，但我们发现，现有最先进的VQA模型学到的多模态表示对此任务的性能贡献仅略微有限。这些结果让人质疑VQA基准测试的成功表现是否表示一般视觉语义推理能力的假设。
URL
https://arxiv.org/abs/1806.00857
