Abstract
Most text-video retrieval methods utilize the text-image pre-trained CLIP as a backbone, incorporating complex modules that result in high computational overhead. As a result, many studies focus on efficient fine-tuning. The primary challenge in efficient adaption arises from the inherent differences between image and video modalities. Each sampled video frame must be processed by the image encoder independently, which increases complexity and complicates practical deployment. Although existing efficient methods fine-tune with small trainable parameters, they still incur high inference costs due to the large token number. In this work, we argue that temporal redundancy significantly contributes to the model's high complexity due to the repeated information in consecutive frames. Existing token compression methods for image models fail to solve the unique challenges, as they overlook temporal redundancy across frames. To tackle these problems, we propose Temporal Token Merging (TempMe) to reduce temporal redundancy. Specifically, we introduce a progressive multi-granularity framework. By gradually combining neighboring clips, we merge temporal tokens across different frames and learn video-level features, leading to lower complexity and better performance. Extensive experiments validate the superiority of our TempMe. Compared to previous efficient text-video retrieval methods, TempMe significantly reduces output tokens by 95% and GFLOPs by 51%, while achieving a 1.8X speedup and a 4.4% R-Sum improvement. Additionally, TempMe exhibits robust generalization capabilities by integrating effectively with both efficient and full fine-tuning methods. With full fine-tuning, TempMe achieves a significant 7.9% R-Sum improvement, trains 1.57X faster, and utilizes 75.2% GPU memory usage. Our code will be released.
Abstract (translated)
大多数文本视频检索方法都利用预训练的CLIP作为骨干，并包含复杂的模块，导致计算开销较高。因此，许多研究关注高效调整。高效适应的主要挑战来自于图像和视频模态固有的差异。每个随机视频帧都必须由图像编码器独立处理，这增加了复杂性并使得实际部署更加复杂。尽管现有的高效方法通过较小的训练参数进行微调，但由于具有较大的标记数量，它们仍然导致较高的推理成本。在本文中，我们认为时间冗余对模型由于连续帧间信息的重复而导致的复杂度高有重大影响。现有的图像模型标记压缩方法未能解决这一独特挑战，因为它们忽视了帧间的时间冗余。为了解决这些问题，我们提出了 Temporal Token Merging（TempMe）来减少时间冗余。具体来说，我们引入了一个渐进多粒度框架。通过逐渐合并相邻片段，我们跨越不同帧合并时间标记，并学习视频级特征，从而降低复杂性并获得更好的性能。大量实验证实了 TempMe 的优越性。与之前的文本视频检索方法相比，TempMe 通过减少输出标记 by 95% 和 GFLOPs by 51% 显著提高了性能。此外，TempMe 通过有效地集成高效的和完全调整方法展示了稳健的泛化能力。在完全调整后，TempMe 的 R-Sum 改进达到了 7.9%，训练速度达到 1.57X，并且使用了 75.2% 的GPU内存。我们的代码将发布。
URL
https://arxiv.org/abs/2409.01156
