Abstract
Video captioning which automatically translates video clips into natural language sentences is a very important task in computer vision. By virtue of recent deep learning technologies, e.g., convolutional neural networks (CNNs) and recurrent neural networks (RNNs), video captioning has made great progress. However, learning an effective mapping from visual sequence space to language space is still a challenging problem. In this paper, we propose a Multimodal Memory Model (M3) to describe videos, which builds a visual and textual shared memory to model the long-term visual-textual dependency and further guide global visual attention on described targets. Specifically, the proposed M3 attaches an external memory to store and retrieve both visual and textual contents by interacting with video and sentence with multiple read and write operations. First, text representation in the Long Short-Term Memory (LSTM) based text decoder is written into the memory, and the memory contents will be read out to guide an attention to select related visual targets. Then, the selected visual information is written into the memory, which will be further read out to the text decoder. To evaluate the proposed model, we perform experiments on two publicly benchmark datasets: MSVD and MSR-VTT. The experimental results demonstrate that our method outperforms the state-of-theart methods in terms of BLEU and METEOR.
Abstract (translated)
自动将视频剪辑转换为自然语言句子的视频字幕是计算机视觉中非常重要的任务。凭借近来的深度学习技术，例如卷积神经网络（CNN）和递归神经网络（RNN），视频字幕已经取得了巨大的进步。然而，从视觉序列空间到语言空间学习有效的映射仍然是一个具有挑战性的问题。在本文中，我们提出了一个多模态记忆模型（M3）来描述视频，这些模型构建了一个视觉和文本共享记忆，以建立长期的视觉文本依赖关系，并进一步引导全球视觉关注所描述的目标。具体而言，所提出的M3附加外部存储器以通过与多个读取和写入操作与视频和句子交互来存储和检索视觉和文本内容。首先，基于长短期存储器（LSTM）的文本解码器中的文本表示被写入存储器，并且存储器内容将被读出以引导关注选择相关的可视目标。然后，所选择的视觉信息被写入到存储器中，该信息将被进一步读出到文本解码器。为了评估所提出的模型，我们在两个公开基准数据集上执行实验：MSVD和MSR-VTT。实验结果表明，我们的方法在BLEU和METEOR方面优于现有方法。
URL
https://arxiv.org/abs/1611.05592
