Abstract
Video classification methods often divide the video into short clips, do inference on these clips independently, and then aggregate these predictions to generate the final classification result. Treating these highly-correlated clips as independent both ignores the temporal structure of the signal and carries a large computational cost: the model must process each clip from scratch. To reduce this cost, recent efforts have focused on designing more efficient clip-level network architectures. Less attention, however, has been paid to the overall framework, including how to benefit from correlations between neighboring clips and improving the aggregation strategy itself. In this paper we leverage the correlation between adjacent video clips to address the problem of computational cost efficiency in video classification at the aggregation stage. More specifically, given a clip feature representation, the problem of computing next clip's representation becomes much easier. We propose a novel recurrent architecture called FASTER for video-level classification, that combines high quality, expensive representations of clips, that capture the action in detail, and lightweight representations, which capture scene changes in the video and avoid redundant computation. We also propose a novel processing unit to learn integration of clip-level representations, as well as their temporal structure. We call this unit FAST-GRU, as it is based on the Gated Recurrent Unit (GRU). The proposed framework achieves significantly better FLOPs vs. accuracy trade-off at inference time. Compared to existing approaches, our proposed framework reduces the FLOPs by more than 10x while maintaining similar accuracy across popular datasets, such as Kinetics, UCF101 and HMDB51.
Abstract (translated)
视频分类方法通常将视频分成短片段，对这些片段进行独立推理，然后将这些预测进行汇总，生成最终的分类结果。将这些高度相关的片段视为独立片段，既忽略了信号的时间结构，又带来了巨大的计算成本：模型必须从头开始处理每个片段。为了降低这一成本，最近的工作重点是设计更高效的剪辑级网络架构。然而，人们对整体框架的关注较少，包括如何从相邻片段之间的相关性中获益以及改进聚合策略本身。本文利用相邻视频片段之间的相关性来解决聚集阶段视频分类的计算成本效率问题。更具体地说，给定一个剪辑特征表示，计算下一个剪辑的表示的问题变得更容易了。我们提出了一种新的循环体系结构，称为“更快的视频级别分类”，它结合了高质量、昂贵的剪辑表示，能够详细捕获动作，以及轻量级表示，能够捕获视频中的场景变化并避免冗余计算。我们还提出了一个新的处理单元来学习剪辑级表示的集成，以及它们的时间结构。我们称这个单元为fast-gru，因为它是基于门控循环单元（gru）。所提出的框架在推理时实现了明显更好的故障与准确度权衡。与现有的方法相比，我们提出的框架减少了10倍以上的故障，同时保持了相同的准确性，在流行的数据集，如动力学，UCF101和HMDB51。
URL
https://arxiv.org/abs/1906.04226
