Abstract
The quest for algorithms that enable cognitive abilities is an important part of machine learning. A common trait in many recently investigated cognitive-like tasks is that they take into account different data modalities, such as visual and textual input. In this paper we propose a novel and generally applicable form of attention mechanism that learns high-order correlations between various data modalities. We show that high-order correlations effectively direct the appropriate attention to the relevant elements in the different data modalities that are required to solve the joint task. We demonstrate the effectiveness of our high-order attention mechanism on the task of visual question answering (VQA), where we achieve state-of-the-art performance on the standard VQA dataset.
Abstract (translated)
追求认知能力的算法是机器学习的重要组成部分。许多最近研究的认知类任务中的一个共同特征是它们考虑了不同的数据模态，例如视觉和文本输入。在本文中，我们提出了一种新颖且普遍适用的注意机制形式，用于了解各种数据模式之间的高阶相关性。我们表明，高阶相关有效地指导了解决联合任务所需的不同数据模式中的相关要素。我们证明了我们的高阶关注机制对视觉问答（VQA）任务的有效性，我们在标准VQA数据集上实现了最先进的性能。
URL
https://arxiv.org/abs/1711.04323
