Abstract
Taking inspiration from physical motion, we present a new self-supervised dynamics learning strategy for videos: Video Time-Differentiation for Instance Discrimination (ViDiDi). ViDiDi is a simple and data-efficient strategy, readily applicable to existing self-supervised video representation learning frameworks based on instance discrimination. At its core, ViDiDi observes different aspects of a video through various orders of temporal derivatives of its frame sequence. These derivatives, along with the original frames, support the Taylor series expansion of the underlying continuous dynamics at discrete times, where higher-order derivatives emphasize higher-order motion features. ViDiDi learns a single neural network that encodes a video and its temporal derivatives into consistent embeddings following a balanced alternating learning algorithm. By learning consistent representations for original frames and derivatives, the encoder is steered to emphasize motion features over static backgrounds and uncover the hidden dynamics in original frames. Hence, video representations are better separated by dynamic features. We integrate ViDiDi into existing instance discrimination frameworks (VICReg, BYOL, and SimCLR) for pretraining on UCF101 or Kinetics and test on standard benchmarks including video retrieval, action recognition, and action detection. The performances are enhanced by a significant margin without the need for large models or extensive datasets.
Abstract (translated)
借鉴物理运动灵感的想法，我们提出了一个新的自监督动态学习策略：视频实例区分（ViDiDi）。ViDiDi是一种简单且数据有效的策略，可以迅速应用于基于实例歧视的自监督视频表示学习框架。在其核心，ViDiDi通过观察帧序列的各个时间导数来关注视频的不同方面。这些导数与原始帧一起支持离散时间下的连续动态的泰勒级数展开。高阶导数突显了更高阶的运动特征。ViDiDi通过一种平衡交替学习算法学习一个视频及其时间导数的一一对应的可视化表示。通过学习原始帧和导数的 consistent representations，编码器被引导强调动态特征，揭示原始帧中的隐藏动态。因此，视频表示更好地通过动态特征进行区分。我们将ViDiDi集成到现有的实例歧视框架（VICReg，BYOL和SimCLR）中，用于对UCF101或Kinetics进行预训练，并在包括视频检索、动作识别和动作检测的标准基准上进行测试。性能通过不需要大型模型或大量数据而显著提高。
URL
https://arxiv.org/abs/2409.02371
