Abstract
In recent text-video retrieval, the use of additional captions from vision-language models has shown promising effects on the performance. However, existing models using additional captions often have struggled to capture the rich semantics, including temporal changes, inherent in the video. In addition, incorrect information caused by generative models can lead to inaccurate retrieval. To address these issues, we propose a new framework, Narrating the Video (NarVid), which strategically leverages the comprehensive information available from frame-level captions, the narration. The proposed NarVid exploits narration in multiple ways: 1) feature enhancement through cross-modal interactions between narration and video, 2) query-aware adaptive filtering to suppress irrelevant or incorrect information, 3) dual-modal matching score by adding query-video similarity and query-narration similarity, and 4) hard-negative loss to learn discriminative features from multiple perspectives using the two similarities from different views. Experimental results demonstrate that NarVid achieves state-of-the-art performance on various benchmark datasets.
Abstract (translated)
在最近的文本-视频检索领域，使用视觉语言模型生成的附加字幕已经显示出对性能有积极的影响。然而，现有的利用这些额外字幕的模型往往难以捕捉到视频中固有的丰富语义信息，包括时间变化等因素。此外，由生成模型造成的错误信息会导致检索不准确。为了解决这些问题，我们提出了一种新的框架——叙述视频（NarVid），该框架战略性地利用了从每一帧级别的描述中获得的全面信息，即“叙述”。提出的NarVid在多个方面充分利用了这种叙述：1）通过叙述与视频之间的跨模态交互增强特征；2）基于查询的自适应过滤以抑制无关或错误的信息；3）双模匹配得分，通过将查询-视频相似度和查询-叙述相似度相加来计算；4）利用这两种相似性从不同视角学习区分特性，并引入难例损失。实验结果表明，NarVid在各种基准数据集上达到了最先进的性能水平。
URL
https://arxiv.org/abs/2503.05186
