Abstract
In this paper, the problem of describing visual contents of a video sequence with natural language is addressed. Unlike previous video captioning work mainly exploiting the cues of video contents to make a language description, we propose a reconstruction network (RecNet) in a novel encoder-decoder-reconstructor architecture, which leverages both forward (video to sentence) and backward (sentence to video) flows for video captioning. Specifically, the encoder-decoder component makes use of the forward flow to produce a sentence description based on the encoded video semantic features. Two types of reconstructors are subsequently proposed to employ the backward flow and reproduce the video features from local and global perspectives, respectively, capitalizing on the hidden state sequence generated by the decoder. Moreover, in order to make a comprehensive reconstruction of the video features, we propose to fuse the two types of reconstructors together. The generation loss yielded by the encoder-decoder component and the reconstruction loss introduced by the reconstructor are jointly cast into training the proposed RecNet in an end-to-end fashion. Furthermore, the RecNet is fine-tuned by CIDEr optimization via reinforcement learning, which significantly boosts the captioning performance. Experimental results on benchmark datasets demonstrate that the proposed reconstructor can boost the performance of video captioning consistently.
Abstract (translated)
本文讨论了用自然语言描述视频序列的视觉内容的问题。与以往的视频字幕工作不同，我们主要利用视频内容的线索进行语言描述，提出了一种新的编码器-解码器-重构器结构中的重构网络（recnet），它利用前向（视频到句子）和后向（句子到视频）流进行视频字幕。具体地说，编码器-解码器组件利用前向流生成基于编码视频语义特征的句子描述。随后提出了两种类型的重构器，利用解码器产生的隐藏状态序列，分别从局部和全局两个角度利用倒流和再现视频特征。此外，为了对视频特征进行全面的重构，我们建议将这两种重构器融合在一起。编码器-解码器组件产生的生成损耗和重构器引入的重构损耗共同构成了端到端的训练方法。此外，通过强化学习，通过苹果酒优化对recnet进行微调，显著提高了字幕性能。在基准数据集上的实验结果表明，所提出的重构器能够始终如一地提高视频字幕的性能。
URL
https://arxiv.org/abs/1906.01452
