Abstract
Top-down visual attention mechanisms have been used extensively in image captioning and visual question answering (VQA) to enable deeper image understanding through fine-grained analysis and even multiple steps of reasoning. In this work, we propose a combined bottom-up and top-down attention mechanism that enables attention to be calculated at the level of objects and other salient image regions. This is the natural basis for attention to be considered. Within our approach, the bottom-up mechanism (based on Faster R-CNN) proposes image regions, each with an associated feature vector, while the top-down mechanism determines feature weightings. Applying this approach to image captioning, our results on the MSCOCO test server establish a new state-of-the-art for the task, achieving CIDEr / SPICE / BLEU-4 scores of 117.9, 21.5 and 36.9, respectively. Demonstrating the broad applicability of the method, applying the same approach to VQA we obtain first place in the 2017 VQA Challenge.
Abstract (translated)
自顶向下的视觉注意机制已广泛用于图像字幕和视觉问题解答（VQA），通过细粒度分析甚至多个推理步骤实现更深入的图像理解。在这项工作中，我们提出了一种自下而上和自上而下的综合注意机制，可以在对象和其他显着图像区域的层面上计算注意力。这是需要考虑注意的自然基础。在我们的方法中，自下而上的机制（基于更快的R-CNN）提出图像区域，每个图像具有相关的特征矢量，而自顶向下机制确定特征权重。将这种方法应用于图像字幕，我们在MSCOCO测试服务器上的结果为该任务建立了新的技术水平，分别实现了117.9,21.5和36.9的CIDEr / SPICE / BLEU-4分数。展示该方法的广泛适用性，将相同的方法应用于VQA，我们将在2017年VQA挑战赛中获得第一名。
URL
https://arxiv.org/abs/1707.07998
