Abstract
Large-scale video-language pretraining enables strong generalization across multimodal tasks but often incurs prohibitive computational costs. Although recent advances in masked visual modeling help mitigate this issue, they still suffer from two fundamental limitations: severe visual information loss under high masking ratios and temporal information leakage caused by inter-frame correlations. To address these challenges, we propose ClusterSTM, a Cluster-Wise Spatio-Temporal Masking strategy for efficient video-language pretraining. ClusterSTM first performs intra-frame clustering to partition visual tokens into multiple semantically independent clusters, then conducts cluster-wise masking by retaining the token with the highest temporal density within each cluster. Our masking strategy ensure that the retained tokens capture holistic video content while exhibit strong temporal correlation. Additionally, we introduce a video-text relevance reconstruction objective that aligns high-level multimodal semantics beyond conventional visual reconstruction. Extensive experiments across multiple benchmarks demonstrate that ClusterSTM achieves superior performance on video-text retrieval, video question answering, and video captioning tasks, establishing a new state-of-the-art among efficient video-language models.
Abstract (translated)
大规模视频-语言预训练虽能实现多模态任务的强泛化能力，但常伴随高昂的计算成本。尽管近期掩码视觉建模的进展有助于缓解此问题，仍存在两个根本局限：高掩码比率下严重的视觉信息损失，以及帧间相关性导致的时间信息泄露。为应对这些挑战，我们提出ClusterSTM——一种面向高效视频-语言预训练的聚类时空掩码策略。该方法首先进行帧内聚类，将视觉标记划分为多个语义独立的聚类；随后执行聚类掩码，在每个聚类中保留时间密度最高的标记。此策略确保保留的标记既能捕获视频整体内容，又具备强时间相关性。此外，我们引入视频-文本相关性重建目标，超越传统视觉重建以对齐高层多模态语义。多项基准实验表明，ClusterSTM在视频-文本检索、视频问答和视频描述任务中均取得卓越性能，在高效视频-语言模型中建立了新的最先进水平。
URL
https://arxiv.org/abs/2603.22953
