Abstract
Visual Question answering is a challenging problem requiring a combination of concepts from Computer Vision and Natural Language Processing. Most existing approaches use a two streams strategy, computing image and question features that are consequently merged using a variety of techniques. Nonetheless, very few rely on higher level image representations, which allow to capture semantic and spatial relationships. In this paper, we propose a novel graph-based approach for Visual Question Answering. Our method combines a graph learner module, which learns a question specific graph representation of the input image, with the recent concept of graph convolutions, aiming to learn image representations that capture question specific interactions. We test our approach on the VQA v2 dataset using a simple baseline architecture enhanced by the proposed graph learner module. We obtain state of the art results with 65.77% accuracy and demonstrate the interpretability of the proposed method.
Abstract (translated)
Visual Question Answering是一个具有挑战性的问题，需要结合计算机视觉和自然语言处理的概念。大多数现有的方法使用双流策略，计算图像和问题特征，因此使用各种技术合并。尽管如此，很少依赖更高级别的图像表示，这允许捕获语义和空间关系。在本文中，我们提出了一种新颖的基于图形的视觉问答方法。我们的方法结合了一个图形学习器模块，该模块学习输入图像的问题特定图形表示与最近的图形卷积概念，旨在学习捕获问题特定交互的图像表示。我们在VQA v2数据集上使用由图形学习器模块增强的简单基线架构来测试我们的方法。我们以65.77％的准确度获得了最新的结果，并证明了所提出方法的可解释性。
URL
https://arxiv.org/abs/1806.07243
