Abstract
Masked video modeling~(MVM) has emerged as a highly effective pre-training strategy for visual foundation models, whereby the model reconstructs masked spatiotemporal tokens using information from visible tokens. However, a key challenge in such approaches lies in selecting an appropriate masking strategy. Previous studies have explored predefined masking techniques, including random and tube-based masking, as well as approaches that leverage key motion priors, optical flow and semantic cues from externally pre-trained models. In this work, we introduce a novel and generalizable Trajectory-Aware Adaptive Token Sampler (TATS), which models the motion dynamics of tokens and can be seamlessly integrated into the masked autoencoder (MAE) framework to select motion-centric tokens in videos. Additionally, we propose a unified training strategy that enables joint optimization of both MAE and TATS from scratch using Proximal Policy Optimization (PPO). We show that our model allows for aggressive masking without compromising performance on the downstream task of action recognition while also ensuring that the pre-training remains memory efficient. Extensive experiments of the proposed approach across four benchmarks, including Something-Something v2, Kinetics-400, UCF101, and HMDB51, demonstrate the effectiveness, transferability, generalization, and efficiency of our work compared to other state-of-the-art methods.
Abstract (translated)
遮罩视频建模（Masked Video Modeling，MVM）已成为视觉基础模型预训练的高效策略之一。在这种方法中，通过可见标记的信息来重构被掩盖的空间-时间标记。然而，在这种技术中的关键挑战是如何选择合适的掩码策略。先前的研究探索了多种预先定义的掩码技术，包括随机和基于管状的方法，并且也考虑了从外部预训练模型中提取的关键运动先验、光流以及语义线索。 在此项工作中，我们引入了一种新颖且具普适性的轨迹感知自适应标记采样器（Trajectory-Aware Adaptive Token Sampler, TATS），该方法能够建模标记的动态性，并可无缝集成到遮罩自动编码器（MAE）框架中以选择视频中的运动核心标记。此外，我们提出了一种统一的训练策略，利用近端策略优化（Proximal Policy Optimization，PPO）从零开始联合优化 MAE 和 TATS。我们的模型允许在不损害动作识别下游任务性能的情况下进行激进的掩码操作，并且还能确保预训练过程保持内存效率。 我们在四个基准数据集上的广泛实验——包括Something-Something v2、Kinetics-400、UCF101和HMDB51，表明我们提出的方法在有效性、可迁移性、泛化能力和效率方面优于其他最先进的方法。
URL
https://arxiv.org/abs/2505.08561
