Abstract
In this paper, we propose an efficient and high-performance method for partially relevant video retrieval (PRVR), which aims to retrieve untrimmed long videos that contain at least one relevant moment to the input text query. In terms of both efficiency and performance, the overlooked bottleneck of previous studies is the visual encoding of dense frames. This guides researchers to choose lightweight visual backbones, yielding sub-optimal retrieval performance due to their limited capabilities of learned visual representations. However, it is undesirable to simply replace them with high-performance large-scale vision-and-language models (VLMs) due to their low efficiency. To address these issues, instead of dense frames, we focus on super images, which are created by rearranging the video frames in a $N \times N$ grid layout. This reduces the number of visual encodings to $\frac{1}{N^2}$ and compensates for the low efficiency of large-scale VLMs, allowing us to adopt them as powerful encoders. Surprisingly, we discover that with a simple query-image attention trick, VLMs generalize well to super images effectively and demonstrate promising zero-shot performance against SOTA methods efficiently. In addition, we propose a fine-tuning approach by incorporating a few trainable modules into the VLM backbones. The experimental results demonstrate that our approaches efficiently achieve the best performance on ActivityNet Captions and TVR.
Abstract (translated)
在本文中，我们提出了一种高效且高性能的部分相关视频检索（PRVR）方法，旨在检索输入文本查询中包含至少一个相关时刻的未剪辑长视频。在效率和性能方面，之前的研究被忽视的一个瓶颈是密帧的视觉编码。这使得研究者选择轻量级的视觉骨干，但由于其学习到的视觉表示能力有限，导致检索性能低于他们的能力。然而，简单地用高性能的大规模视觉与语言模型（VLMs）替换它们并不理想，因为它们的效率太低了。为了应对这些问题，我们关注超图像，这是通过将视频帧按照 $N \times N$ 的网格布局重新排列来创建的。这减少了视觉编码的数量至 $\frac{1}{N^2}$，并弥补了大规模 VLMs 的低效率，使我们可以将它们用作强大的编码器。令人惊讶的是，我们发现，通过一个简单的查询图像关注技巧，VLMs 很好地向超图像进行扩展，并高效地对抗了目前的最优方法。此外，我们通过将几个可训练的模块集成到 VLM 骨干网络中，提出了一种微调方法。实验结果表明，我们的方法在活动网络捕捉和 TVR 上实现了最佳性能。
URL
https://arxiv.org/abs/2312.00414
