Abstract
Visual question answering (VQA) is of significant interest due to its potential to be a strong test of image understanding systems and to probe the connection between language and vision. Despite much recent progress, general VQA is far from a solved problem. In this paper, we focus on the VQA multiple-choice task, and provide some good practices for designing an effective VQA model that can capture language-vision interactions and perform joint reasoning. We explore mechanisms of incorporating part-of-speech (POS) tag guided attention, convolutional n-grams, triplet attention interactions between the image, question and candidate answer, and structured learning for triplets based on image-question pairs. We evaluate our models on two popular datasets: Visual7W and VQA Real Multiple Choice. Our final model achieves the state-of-the-art performance of 68.2% on Visual7W, and a very competitive performance of 69.6% on the test-standard split of VQA Real Multiple Choice.
Abstract (translated)
视觉问题回答（VQA）具有重要意义，因为它有潜力成为图像理解系统的强大测试，并探讨语言与视觉之间的联系。尽管最近取得了很多进展，但一般的VQA还远未解决。在本文中，我们将重点放在VQA多项选择任务上，并提供一些设计有效VQA模型的良好实践，可以捕获语言视觉交互并执行联合推理。我们探索了基于图像问题对的词性标注引导注意，卷积n-gram，图像，问题和候选答案之间的三重关注交互以及三元组结构化学习的机制。我们在两个流行数据集上评估我们的模型：Visual7W和VQA Real Multiple Choice。我们的最终模型在Visual7W上实现了68.2％的最新性能，并且在VQA Real Multiple Choice的测试标准分割上实现了69.6％的非常有竞争力的性能。
URL
https://arxiv.org/abs/1801.07853
