Abstract
The key of the text-to-video retrieval (TVR) task lies in learning the unique similarity between each pair of text (consisting of words) and video (consisting of audio and image frames) representations. However, some problems exist in the representation alignment of video and text, such as a text, and further each word, are of different importance for video frames. Besides, audio usually carries additional or critical information for TVR in the case that frames carry little valid information. Therefore, in TVR task, multi-granularity representation of text, including whole sentence and every word, and the modal of audio are salutary which are underutilized in most existing works. To address this, we propose a novel multi-granularity feature interaction module called MGFI, consisting of text-frame and word-frame, for video-text representations alignment. Moreover, we introduce a cross-modal feature interaction module of audio and text called CMFI to solve the problem of insufficient expression of frames in the video. Experiments on benchmark datasets such as MSR-VTT, MSVD, DiDeMo show that the proposed method outperforms the existing state-of-the-art methods.
Abstract (translated)
文本转视频检索（TVR）任务的关键在于学习每对文本（由单词组成）和视频（由音频和图像帧组成）表示之间的独特相似性。然而，在文本和视频表示对齐的过程中存在一些问题，例如文本和每个单词的重要性不同，以及在视频帧中携带的有效信息通常较少。此外，音频通常携带与TVR无关的额外或关键信息。因此，在TVR任务中，多粒度文本表示，包括整个句子和每个单词，以及音频的模态，在大多数现有作品中都被低估了。为了解决这个问题，我们提出了一个名为MGFI的多粒度特征交互模块，用于视频文本表示对齐。此外，我们还引入了一个跨模态特征交互模块来解决视频中帧表达不足的问题。在基准数据集如MSR-VTT、MSVD和DiDeMo上进行实验，结果表明，与现有最先进的方法相比，所提出的方法表现出优异的性能。
URL
https://arxiv.org/abs/2407.12798
