Abstract
Text-Video Retrieval (TVR) aims to align relevant video content with natural language queries. To date, most state-of-the-art TVR methods learn image-to-video transfer learning based on large-scale pre-trained visionlanguage models (e.g., CLIP). However, fully fine-tuning these pre-trained models for TVR incurs prohibitively expensive computation costs. To this end, we propose to conduct efficient text-video Retrieval with a sparse-andcorrelated AdaPter (RAP), i.e., fine-tuning the pre-trained model with a few parameterized layers. To accommodate the text-video scenario, we equip our RAP with two indispensable characteristics: temporal sparsity and correlation. Specifically, we propose a low-rank modulation module to refine the per-image features from the frozen CLIP backbone, which accentuates salient frames within the video features while alleviating temporal redundancy. Besides, we introduce an asynchronous self-attention mechanism that first selects the top responsive visual patches and augments the correlation modeling between them with learnable temporal and patch offsets. Extensive experiments on four TVR datasets demonstrate that RAP achieves superior or comparable performance compared to the fully fine-tuned counterpart and other parameter-efficient fine-tuning methods.
Abstract (translated)
文本-视频检索（TVR）旨在将相关的视频内容与自然语言查询对齐。目前，最先进的TVR方法基于大规模预训练的视觉语言模型（例如，CLIP）学习图像到视频的迁移学习。然而，完全对预训练模型进行TVR会带来不必要的计算成本。为此，我们提出了一种高效的文本-视频检索方法与稀疏相关AdaPter（RAP），即在预训练模型上进行少量参数化层的学习。为了适应文本-视频场景，我们为我们的RAP配备了两个不可或缺的特征：时间稀疏性和相关性。具体来说，我们提出了一个低秩调制模块来微调预训练的CLIP骨干网络中的每张图片特征，这使得视频特征中的显眼帧更加突出，同时减轻了时间冗余。此外，我们还引入了一个异步自注意机制，它首先选择最相关的视觉补丁，并通过可学习的时间和补丁偏移增强它们之间的相关建模。在四个TVR数据集上的大量实验证明，RAP实现了与完全参数化方法相比优越或可比较的性能，以及其他参数高效的 fine-tuning 方法。
URL
https://arxiv.org/abs/2405.19465
