Abstract
Two-stream architecture have shown strong performance in video classification task. The key idea is to learn spatio-temporal features by fusing convolutional networks spatially and temporally. However, there are some problems within such architecture. First, it relies on optical flow to model temporal information, which are often expensive to compute and store. Second, it has limited ability to capture details and local context information for video data. Third, it lacks explicit semantic guidance that greatly decrease the classification performance. In this paper, we proposed a new two-stream based deep framework for video classification to discover spatial and temporal information only from RGB frames, moreover, the multi-scale pyramid attention (MPA) layer and the semantic adversarial learning (SAL) module is introduced and integrated in our framework. The MPA enables the network capturing global and local feature to generate a comprehensive representation for video, and the SAL can make this representation gradually approximate to the real video semantics in an adversarial manner. Experimental results on two public benchmarks demonstrate our proposed methods achieves state-of-the-art results on standard video datasets.
Abstract (translated)
双流体系结构在视频分类任务中表现出很强的性能。其核心思想是通过时空融合卷积网络来学习时空特征。然而，这种体系结构中存在一些问题。首先，它依靠光流来模拟时间信息，而时间信息的计算和存储通常很昂贵。第二，它捕获视频数据的细节和本地上下文信息的能力有限。第三，它缺乏明确的语义指导，大大降低了分类性能。本文提出了一种新的基于两流的视频分类深度框架，该框架只从RGB帧中发现时空信息，并引入了多尺度金字塔注意层（MPA）和语义对抗学习（SAL）模块，并将其集成到该框架中。MPA使网络捕获的全局和局部特征生成一个全面的视频表示，SAL可以使这种表示逐渐接近于真实的视频语义。两个公共基准的实验结果表明，我们提出的方法在标准视频数据集上达到了最先进的结果。
URL
https://arxiv.org/abs/1903.02155
