Abstract
Several video understanding tasks, such as natural language temporal video grounding, temporal activity localization, and audio description generation, require "temporally dense" reasoning over frames sampled at high temporal resolution. However, computing frame-level features for these tasks is computationally expensive given the temporal resolution requirements. In this paper, we make three contributions to reduce the cost of computing features for temporally dense tasks. First, we introduce a vision transformer (ViT) architecture, dubbed ResidualViT, that leverages the large temporal redundancy in videos to efficiently compute temporally dense frame-level features. Our architecture incorporates (i) learnable residual connections that ensure temporal consistency across consecutive frames and (ii) a token reduction module that enhances processing speed by selectively discarding temporally redundant information while reusing weights of a pretrained foundation model. Second, we propose a lightweight distillation strategy to approximate the frame-level features of the original foundation model. Finally, we evaluate our approach across four tasks and five datasets, in both zero-shot and fully supervised settings, demonstrating significant reductions in computational cost (up to 60%) and improvements in inference speed (up to 2.5x faster), all while closely approximating the accuracy of the original foundation model.
Abstract (translated)
几项视频理解任务，如自然语言时间定位、时序活动定位和音频描述生成，需要对以高时间分辨率抽取的帧进行“密集的时间”推理。然而，由于时间分辨率的要求，为这些任务计算每帧特征在计算上是昂贵的。本文提出了三项贡献来降低密集时间任务中特征计算的成本。首先，我们引入了一种视觉变换器（ViT）架构，称为ResidualViT，它利用视频中的大量时间冗余性高效地计算出密集的时间帧级特征。我们的架构结合了(i)可学习残差连接，以确保连续帧之间的时间一致性；(ii)一个标记减少模块，通过选择性丢弃时间冗余信息并重用预训练基础模型的权重来提高处理速度。其次，我们提出了一种轻量级的知识蒸馏策略，用于逼近原基础模型的每帧特征。最后，我们在四种任务和五个数据集上进行了评估，并在零样本学习和完全监督设置下展示了显著降低计算成本（高达60%）和提升推理速度（快至2.5倍），同时接近原基础模型的准确性。
URL
https://arxiv.org/abs/2509.13255
