Paper Reading AI Learner

Explicit Reasoning over End-to-End Neural Architectures for Visual Question Answering

2018-03-23 17:17:16
Somak Aditya, Yezhou Yang, Chitta Baral

Abstract

Many vision and language tasks require commonsense reasoning beyond data-driven image and natural language processing. Here we adopt Visual Question Answering (VQA) as an example task, where a system is expected to answer a question in natural language about an image. Current state-of-the-art systems attempted to solve the task using deep neural architectures and achieved promising performance. However, the resulting systems are generally opaque and they struggle in understanding questions for which extra knowledge is required. In this paper, we present an explicit reasoning layer on top of a set of penultimate neural network based systems. The reasoning layer enables reasoning and answering questions where additional knowledge is required, and at the same time provides an interpretable interface to the end users. Specifically, the reasoning layer adopts a Probabilistic Soft Logic (PSL) based engine to reason over a basket of inputs: visual relations, the semantic parse of the question, and background ontological knowledge from word2vec and ConceptNet. Experimental analysis of the answers and the key evidential predicates generated on the VQA dataset validate our approach.

Abstract (translated)

除了数据驱动的图像和自然语言处理外,许多视觉和语言任务都需要常识推理。在这里,我们采用视觉问答(VQA)作为示例任务,系统需要用自然语言回答关于图像的问题。当前最先进的系统尝试使用深度神经架构来解决任务,并取得了令人满意的性能。但是,由此产生的系统通常是不透明的,他们很难理解需要额外知识的问题。在本文中,我们在一组倒数第二个基于神经网络的系统之上提出了一个明确的推理层。推理层可以在需要额外知识的情况下推理和回答问题,同时为最终用户提供可解释的界面。具体而言,推理层采用基于概率软逻辑(PSL)的引擎来推理一篮子输入:视觉关系,问题的语义解析以及来自word2vec和ConceptNet的背景知识本体。在VQA数据集上生成的答案和关键证据预测的实验分析验证了我们的方法。

URL

https://arxiv.org/abs/1803.08896

PDF

https://arxiv.org/pdf/1803.08896.pdf


Tags
3D Action Action_Localization Action_Recognition Activity Adversarial Agent Attention Autonomous Bert Boundary_Detection Caption Chat Classification CNN Compressive_Sensing Contour Contrastive_Learning Deep_Learning Denoising Detection Dialog Diffusion Drone Dynamic_Memory_Network Edge_Detection Embedding Embodied Emotion Enhancement Face Face_Detection Face_Recognition Facial_Landmark Few-Shot Gait_Recognition GAN Gaze_Estimation Gesture Gradient_Descent Handwriting Human_Parsing Image_Caption Image_Classification Image_Compression Image_Enhancement Image_Generation Image_Matting Image_Retrieval Inference Inpainting Intelligent_Chip Knowledge Knowledge_Graph Language_Model Matching Medical Memory_Networks Multi_Modal Multi_Task NAS NMT Object_Detection Object_Tracking OCR Ontology Optical_Character Optical_Flow Optimization Person_Re-identification Point_Cloud Portrait_Generation Pose Pose_Estimation Prediction QA Quantitative Quantitative_Finance Quantization Re-identification Recognition Recommendation Reconstruction Regularization Reinforcement_Learning Relation Relation_Extraction Represenation Represenation_Learning Restoration Review RNN Salient Scene_Classification Scene_Generation Scene_Parsing Scene_Text Segmentation Self-Supervised Semantic_Instance_Segmentation Semantic_Segmentation Semi_Global Semi_Supervised Sence_graph Sentiment Sentiment_Classification Sketch SLAM Sparse Speech Speech_Recognition Style_Transfer Summarization Super_Resolution Surveillance Survey Text_Classification Text_Generation Tracking Transfer_Learning Transformer Unsupervised Video_Caption Video_Classification Video_Indexing Video_Prediction Video_Retrieval Visual_Relation VQA Weakly_Supervised Zero-Shot