Abstract
Video retrieval requires aligning visual content with corresponding natural language descriptions. In this paper, we introduce Modality Auxiliary Concepts for Video Retrieval (MAC-VR), a novel approach that leverages modality-specific tags -- automatically extracted from foundation models -- to enhance video retrieval. We propose to align modalities in a latent space, along with learning and aligning auxiliary latent concepts, derived from the features of a video and its corresponding caption. We introduce these auxiliary concepts to improve the alignment of visual and textual latent concepts, and so are able to distinguish concepts from one other. We conduct extensive experiments on five diverse datasets: MSR-VTT, DiDeMo, TGIF, Charades and YouCook2. The experimental results consistently demonstrate that modality-specific tags improve cross-modal alignment, outperforming current state-of-the-art methods across three datasets and performing comparably or better across the other two.
Abstract (translated)
视频检索需要将视觉内容与相应的自然语言描述对齐。在本文中，我们介绍了用于视频检索的模态辅助概念（MAC-VR），这是一种新颖的方法，它利用了从基础模型自动提取的特定于模态的标签来增强视频检索。我们提出了一种方法，在潜在空间中对齐模态，并且通过学习和对齐来自视频及其相应字幕特征派生出的辅助潜在概念来进行这种对齐。我们引入这些辅助概念以改善视觉和文本潜在概念之间的对齐，从而使我们可以区分彼此的概念。 我们在五个多样化的数据集上进行了广泛的实验：MSR-VTT、DiDeMo、TGIF、Charades 和 YouCook2。实验结果一致地表明，特定于模态的标签可以提高跨模态对齐性能，在三个数据集中优于当前最先进的方法，并且在其他两个数据集中表现出色或相当。
URL
https://arxiv.org/abs/2504.01591
