Abstract
In this paper, we introduce Key-Value Memory Networks to a multimodal setting and a novel key-addressing mechanism to deal with sequence-to-sequence models. The proposed model naturally decomposes the problem of video captioning into vision and language segments, dealing with them as key-value pairs. More specifically, we learn a semantic embedding (v) corresponding to each frame (k) in the video, thereby creating (k, v) memory slots. We propose to find the next step attention weights conditioned on the previous attention distributions for the key-value memory slots in the memory addressing schema. Exploiting this flexibility of the framework, we additionally capture spatial dependencies while mapping from the visual to semantic embedding. Experiments done on the Youtube2Text dataset demonstrate usefulness of recurrent key-addressing, while achieving competitive scores on BLEU@4, METEOR metrics against state-of-the-art models.
Abstract (translated)
在本文中，我们将关键值存储器网络引入到多模式设置和一种处理序列到序列模型的新型密钥寻址机制中。所提出的模型自然将视频字幕问题分解为视觉和语言片段，将它们作为键值对处理。更具体地说，我们学习了对应于视频中的每个帧（k）的语义嵌入（v），由此创建（k，v）个存储器时隙。我们建议寻找下一步的关注权重，这些权重是以内存寻址模式中的键值内存插槽的先前关注分布为条件的。利用框架的这种灵活性，我们另外捕获了从视觉到语义嵌入的空间依赖关系。在Youtube2Text数据集上进行的实验证明了经常使用的键盘寻址的有用性，同时在BLEU @ 4上获得了竞争力得分，METEOR指标与最先进的模型相比较。
URL
https://arxiv.org/abs/1611.06492
