Abstract
Exploiting the temporal dependency among video frames or subshots is very important for the task of video summarization. Practically, RNN is good at temporal dependency modeling, and has achieved overwhelming performance in many video-based tasks, such as video captioning and classification. However, RNN is not capable enough to handle the video summarization task, since traditional RNNs, including LSTM, can only deal with short videos, while the videos in the summarization task are usually in longer duration. To address this problem, we propose a hierarchical recurrent neural network for video summarization, called H-RNN in this paper. Specifically, it has two layers, where the first layer is utilized to encode short video subshots cut from the original video, and the final hidden state of each subshot is input to the second layer for calculating its confidence to be a key subshot. Compared to traditional RNNs, H-RNN is more suitable to video summarization, since it can exploit long temporal dependency among frames, meanwhile, the computation operations are significantly lessened. The results on two popular datasets, including the Combined dataset and VTW dataset, have demonstrated that the proposed H-RNN outperforms the state-of-the-arts.
Abstract (translated)
利用视频帧或子图像之间的时间依赖性对视频摘要的任务非常重要。实际上，RNN擅长于时间依赖性建模，并在许多基于视频的任务（如视频字幕和分类）中取得了压倒性的性能。然而，RNN不足以处理视频摘要任务，因为传统的RNN（包括LSTM）只能处理短视频，而摘要任务中的视频通常持续时间较长。为了解决这个问题，本文提出了一种用于视频总结的层次递归神经网络H-RNN。具体地说，它有两层，第一层用于编码从原始视频剪切的短视频子照片，每个子照片的最终隐藏状态输入到第二层，以计算其作为关键子照片的置信度。与传统的RNN相比，H-RNN更适合视频摘要，因为它可以利用帧间的长时间依赖性，同时大大减少了计算量。两个流行数据集（包括组合数据集和VTW数据集）的结果表明，所提出的H-RNN优于现有技术。
URL
https://arxiv.org/abs/1904.12251
