Abstract
Self-supervised approaches for video have shown impressive results in video understanding tasks. However, unlike early works that leverage temporal self-supervision, current state-of-the-art methods primarily rely on tasks from the image domain (e.g., contrastive learning) that do not explicitly promote the learning of temporal features. We identify two factors that limit existing temporal self-supervision: 1) tasks are too simple, resulting in saturated training performance, and 2) we uncover shortcuts based on local appearance statistics that hinder the learning of high-level features. To address these issues, we propose 1) a more challenging reformulation of temporal self-supervision as frame-level (rather than clip-level) recognition tasks and 2) an effective augmentation strategy to mitigate shortcuts. Our model extends a representation of single video frames, pre-trained through contrastive learning, with a transformer that we train through temporal self-supervision. We demonstrate experimentally that our more challenging frame-level task formulations and the removal of shortcuts drastically improve the quality of features learned through temporal self-supervision. The generalization capability of our self-supervised video method is evidenced by its state-of-the-art performance in a wide range of high-level semantic tasks, including video retrieval, action classification, and video attribute recognition (such as object and scene identification), as well as low-level temporal correspondence tasks like video object segmentation and pose tracking. Additionally, we show that the video representations learned through our method exhibit increased robustness to the input perturbations.
Abstract (translated)
自监督方法在视频理解任务中已经取得了令人印象深刻的成果。然而，与利用时域自监督的早期工作不同，目前最先进的视频方法主要依赖于来自图像领域的任务（例如对比学习），而没有明确促进学习时域特征。我们发现现有时域自监督的两个限制因素：1）任务过于简单，导致训练效果饱和；2）我们基于局部外观统计学发现了短路，从而阻碍了高层次特征的学习。为了应对这些问题，我们提出了以下两个方案：1）将时域自监督重新定义为帧级（而不是片段级）识别任务，2）采用有效的增强策略来缓解短路。我们的模型通过通过时域自监督预训练的单个视频帧，加上我们通过时域自监督训练的Transformer，来扩展预训练的表示。我们通过实验证明了， our more challenging frame-level task formulations and the removal of shortcuts significantly improve the quality of features learned through temporal self-supervision. 我们模型的自监督视频方法的泛化能力体现在其在广泛的先进语义任务中的最佳性能，包括视频检索、动作分类和视频属性识别（如物体和场景识别），以及低级时序对应任务，如视频物体分割和姿态跟踪。此外，我们还证明了通过我们的方法学习到的视频表示具有对输入扰动的增强鲁棒性。
URL
https://arxiv.org/abs/2312.13008
