Abstract
Many vision and language tasks require commonsense reasoning beyond data-driven image and natural language processing. Here we adopt Visual Question Answering (VQA) as an example task, where a system is expected to answer a question in natural language about an image. Current state-of-the-art systems attempted to solve the task using deep neural architectures and achieved promising performance. However, the resulting systems are generally opaque and they struggle in understanding questions for which extra knowledge is required. In this paper, we present an explicit reasoning layer on top of a set of penultimate neural network based systems. The reasoning layer enables reasoning and answering questions where additional knowledge is required, and at the same time provides an interpretable interface to the end users. Specifically, the reasoning layer adopts a Probabilistic Soft Logic (PSL) based engine to reason over a basket of inputs: visual relations, the semantic parse of the question, and background ontological knowledge from word2vec and ConceptNet. Experimental analysis of the answers and the key evidential predicates generated on the VQA dataset validate our approach.
Abstract (translated)
除了数据驱动的图像和自然语言处理外,许多视觉和语言任务都需要常识推理。在这里,我们采用视觉问答(VQA)作为示例任务,系统需要用自然语言回答关于图像的问题。当前最先进的系统尝试使用深度神经架构来解决任务,并取得了令人满意的性能。但是,由此产生的系统通常是不透明的,他们很难理解需要额外知识的问题。在本文中,我们在一组倒数第二个基于神经网络的系统之上提出了一个明确的推理层。推理层可以在需要额外知识的情况下推理和回答问题,同时为最终用户提供可解释的界面。具体而言,推理层采用基于概率软逻辑(PSL)的引擎来推理一篮子输入:视觉关系,问题的语义解析以及来自word2vec和ConceptNet的背景知识本体。在VQA数据集上生成的答案和关键证据预测的实验分析验证了我们的方法。
URL
https://arxiv.org/abs/1803.08896