Abstract
In recent years, the explosion of web videos makes text-video retrieval increasingly essential and popular for video filtering, recommendation, and search. Text-video retrieval aims to rank relevant text/video higher than irrelevant ones. The core of this task is to precisely measure the cross-modal similarity between texts and videos. Recently, contrastive learning methods have shown promising results for text-video retrieval, most of which focus on the construction of positive and negative pairs to learn text and video representations. Nevertheless, they do not pay enough attention to hard negative pairs and lack the ability to model different levels of semantic similarity. To address these two issues, this paper improves contrastive learning using two novel techniques. First, to exploit hard examples for robust discriminative power, we propose a novel Dual-Modal Attention-Enhanced Module (DMAE) to mine hard negative pairs from textual and visual clues. By further introducing a Negative-aware InfoNCE (NegNCE) loss, we are able to adaptively identify all these hard negatives and explicitly highlight their impacts in the training loss. Second, our work argues that triplet samples can better model fine-grained semantic similarity compared to pairwise samples. We thereby present a new Triplet Partial Margin Contrastive Learning (TPM-CL) module to construct partial order triplet samples by automatically generating fine-grained hard negatives for matched text-video pairs. The proposed TPM-CL designs an adaptive token masking strategy with cross-modal interaction to model subtle semantic differences. Extensive experiments demonstrate that the proposed approach outperforms existing methods on four widely-used text-video retrieval datasets, including MSR-VTT, MSVD, DiDeMo and ActivityNet.
Abstract (translated)
近年来,Web视频的爆炸使得文本-视频检索对于视频过滤器、推荐和搜索变得越来越重要和流行。文本-视频检索的目标是将相关的文本/视频排名高于无关的文本/视频。这项工作的核心在于精确测量文本和视频的跨modal相似性。最近,对比学习方法在文本-视频检索中取得了良好的结果,其中大部分方法专注于构建正交和反交对来学习文本和视频表示。然而,他们并没有足够重视坚固的负对对,并且缺乏建模不同级别的语义相似性的能力。为了解决这些问题,本文使用两个新技术来提高对比学习。首先,利用坚固的示例来增强鲁棒性,我们提出了一种新的双模注意力增强模块(DMAE),以从文本和视觉线索中挖掘坚固的负对对。此外,我们引入了一个负 aware infoNCE(NegNCE)损失,可以自适应地识别所有的坚固的负对对,并明确地突出它们在训练损失中的影响。其次,我们的工作认为三组样本比二组样本更好地模型精细语义相似性。因此,我们提出了一个新的三组 partial margin Contrastive Learning(TPM-CL)模块,以通过自动生成匹配的文本-视频对的精细坚固的负对对来构建 partial order 三组样本。该提出的 TPM-CL 设计了一个自适应的元掩码策略,通过跨modal交互来建模微妙的语义差异。广泛的实验表明,该方法在常用的文本-视频检索数据集上,包括 MSR-VTT、MSVD、DiDeMo 和活动网络中,表现出了比现有方法更好的性能。
URL
https://arxiv.org/abs/2309.11082
