Abstract
Recent studies have made notable progress in video representation learning by transferring image-pretrained models to video tasks, typically with complex temporal modules and video fine-tuning. However, fine-tuning heavy modules may compromise inter-video semantic separability, i.e., the essential ability to distinguish objects across videos. While reducing the tunable parameters hinders their intra-video temporal consistency, which is required for stable representations of the same object within a video. This dilemma indicates a potential trade-off between the intra-video temporal consistency and inter-video semantic separability during image-to-video transfer. To this end, we propose the Consistency-Separability Trade-off Transfer Learning (Co-Settle) framework, which applies a lightweight projection layer on top of the frozen image-pretrained encoder to adjust representation space with a temporal cycle consistency objective and a semantic separability constraint. We further provide a theoretical support showing that the optimized projection yields a better trade-off between the two properties under appropriate conditions. Experiments on eight image-pretrained models demonstrate consistent improvements across multiple levels of video tasks with only five epochs of self-supervised training. The code is available at this https URL.
Abstract (translated)
近期研究通过将图像预训练模型迁移至视频任务，在视频表示学习方面取得了显著进展，通常采用复杂的时间模块和视频微调。然而，微调重型模块可能会损害视频间语义可分性，即跨视频区分对象的基本能力。同时，减少可调参数会阻碍视频内时间一致性，而后者对于视频中同一对象的稳定表示至关重要。这一困境表明，在图像到视频迁移过程中，视频内时间一致性与视频间语义可分性之间存在潜在权衡。为此，我们提出了 Consistency-Separability Trade-off Transfer Learning (Co-Settle) 框架，该框架在冻结的图像预训练编码器之上应用轻量级投影层，通过时间循环一致性目标和语义可分性约束来调整表示空间。我们还提供了理论支持，证明在适当条件下，优化后的投影能在两种属性间取得更好的权衡。在八个图像预训练模型上的实验表明，仅通过五个周期的自监督训练，该方法在多层次视频任务上均实现了一致性提升。代码已公开于此https链接。
URL
https://arxiv.org/abs/2603.26597
