Abstract
Visual question answering (VQA) requires joint comprehension of images and natural language questions, where many questions can't be directly or clearly answered from visual content but require reasoning from structured human knowledge with confirmation from visual content. This paper proposes visual knowledge memory network (VKMN) to address this issue, which seamlessly incorporates structured human knowledge and deep visual features into memory networks in an end-to-end learning framework. Comparing to existing methods for leveraging external knowledge for supporting VQA, this paper stresses more on two missing mechanisms. First is the mechanism for integrating visual contents with knowledge facts. VKMN handles this issue by embedding knowledge triples (subject, relation, target) and deep visual features jointly into the visual knowledge features. Second is the mechanism for handling multiple knowledge facts expanding from question and answer pairs. VKMN stores joint embedding using key-value pair structure in the memory networks so that it is easy to handle multiple facts. Experiments show that the proposed method achieves promising results on both VQA v1.0 and v2.0 benchmarks, while outperforms state-of-the-art methods on the knowledge-reasoning related questions.
Abstract (translated)
视觉问题解答（VQA）需要对图像和自然语言问题进行联合理解，其中许多问题不能直接或清晰地从视觉内容中回答，但需要结构化人类知识的推理和视觉内容的确认。本文提出视觉知识记忆网络（VKMN）来解决这个问题，它将结构化的人类知识和深度视觉特征无缝地结合到端到端学习框架中的记忆网络中。与现有的利用外部知识支持VQA的方法相比，本文强调两个缺失机制。首先是将视觉内容与知识事实相结合的机制。 VKMN通过将知识三元组（主体，关系，目标）和深层视觉特征共同嵌入到视觉知识特征中来处理这个问题。其次是处理从问题和答案对扩大的多个知识事实的机制。 VKMN在存储器网络中存储使用键值对结构的联合嵌入，以便处理多个事实。实验表明，所提出的方法在VQA v1.0和v2.0基准测试中都取得了令人满意的结果，但在知识推理相关问题上优于最先进的方法。
URL
https://arxiv.org/abs/1806.04860
