Abstract
Recently, pre-trained state space models have shown great potential for video classification, which sequentially compresses visual tokens in videos with linear complexity, thereby improving the processing efficiency of video data while maintaining high performance. To apply powerful pre-trained models to downstream tasks, prompt learning is proposed to achieve efficient downstream task adaptation with only a small number of fine-tuned parameters. However, the sequentially compressed visual prompt tokens fail to capture the spatial and temporal contextual information in the video, thus limiting the effective propagation of spatial information within a video frame and temporal information between frames in the state compression model and the extraction of discriminative information. To tackle the above issue, we proposed a State Space Prompting (SSP) method for video understanding, which combines intra-frame and inter-frame prompts to aggregate and propagate key spatiotemporal information in the video. Specifically, an Intra-Frame Gathering (IFG) module is designed to aggregate spatial key information within each frame. Besides, an Inter-Frame Spreading (IFS) module is designed to spread discriminative spatio-temporal information across different frames. By adaptively balancing and compressing key spatio-temporal information within and between frames, our SSP effectively propagates discriminative information in videos in a complementary manner. Extensive experiments on four video benchmark datasets verify that our SSP significantly outperforms existing SOTA methods by 2.76% on average while reducing the overhead of fine-tuning parameters.
Abstract (translated)
最近，预训练状态空间模型在视频分类任务中展现了巨大的潜力。这些模型可以以线性复杂度顺序压缩视频中的视觉标记，从而提高视频数据处理的效率，同时保持高性能。为了将强大的预训练模型应用于下游任务，提出了提示学习方法，通过仅调整少量参数即可实现高效的下游任务适配。然而，顺序压缩的视觉提示令牌无法捕捉到视频中的空间和时间上下文信息，这限制了状态压缩模型中视频帧内空间信息的有效传播以及不同帧之间的时间信息提取，并且影响鉴别性信息的抽取。 为了解决上述问题，我们提出了一个用于视频理解的状态空间提示（SSP）方法。该方法结合了帧内和帧间提示来聚合并传播视频中的关键时空信息。具体来说，设计了一个帧内收集（IFG）模块以在每个帧内部聚集重要的空间信息，并且设计了一个帧间扩散（IFS）模块以便跨不同帧传递区分性时空信息。通过自适应地平衡和压缩帧内的以及帧间的关鍵时-空信息，我们的SSP方法能够互补地传播视频中的鉴别性信息。 在四个视频基准数据集上的大量实验验证了我们提出的SSP方法相较于现有最先进（SOTA）方法平均提高了2.76%，同时减少了微调参数的开销。
URL
https://arxiv.org/abs/2510.12160
