Abstract
Adapting large-scale image-text pre-training models, e.g., CLIP, to the video domain represents the current state-of-the-art for text-video retrieval. The primary approaches involve transferring text-video pairs to a common embedding space and leveraging cross-modal interactions on specific entities for semantic alignment. Though effective, these paradigms entail prohibitive computational costs, leading to inefficient retrieval. To address this, we propose a simple yet effective method, Global-Local Semantic Consistent Learning (GLSCL), which capitalizes on latent shared semantics across modalities for text-video retrieval. Specifically, we introduce a parameter-free global interaction module to explore coarse-grained alignment. Then, we devise a shared local interaction module that employs several learnable queries to capture latent semantic concepts for learning fine-grained alignment. Furthermore, an Inter-Consistency Loss (ICL) is devised to accomplish the concept alignment between the visual query and corresponding textual query, and an Intra-Diversity Loss (IDL) is developed to repulse the distribution within visual (textual) queries to generate more discriminative concepts. Extensive experiments on five widely used benchmarks (i.e., MSR-VTT, MSVD, DiDeMo, LSMDC, and ActivityNet) substantiate the superior effectiveness and efficiency of the proposed method. Remarkably, our method achieves comparable performance with SOTA as well as being nearly 220 times faster in terms of computational cost. Code is available at: this https URL.
Abstract (translated)
将大规模图像-文本预训练模型（例如，CLIP）适应视频领域代表了文本视频检索的现有水平。主要方法是将文本-图像对转移到共同的嵌入空间，并利用特定实体上的跨模态相互作用进行语义对齐。尽管这些范式有效，但它们都意味着高昂的计算成本，导致效率低下。为了应对这个问题，我们提出了一个简单而有效的方法，全局局部语义一致学习（GLSCL），利用跨模态的潜在共享语义进行文本-视频检索。具体来说，我们引入了一个参数无关的全局交互模块来探索粗粒度对齐。然后，我们设计了一个共享局部交互模块，使用多个可学习查询来捕捉潜在语义概念，进行细粒度对齐。此外，还设计了互斥一致损失（ICL）和内差异损失（IDL）来实现视觉查询和相应文本查询之间的概念对齐，以及分布排斥损失（ICR）来生成更具区分性的概念。在五个广泛使用的基准（即MSR-VTT，MSVD，DiDeMo，LSMDC和ActivityNet）上的大量实验证实了所提出方法的有效性和高效性。值得注意的是，我们的方法在SOTA上的性能相当，并且在计算成本方面几乎快了220倍。代码可在此处下载：https:// this URL.
URL
https://arxiv.org/abs/2405.12710
