Abstract
Text-Video Retrieval (TVR) methods typically match query-candidate pairs by aligning text and video features in coarse-grained, fine-grained, or combined (coarse-to-fine) manners. However, these frameworks predominantly employ a one(query)-to-one(candidate) alignment paradigm, which struggles to discern nuanced differences among candidates, leading to frequent mismatches. Inspired by Comparative Judgement in human cognitive science, where decisions are made by directly comparing items rather than evaluating them independently, we propose TokenBinder. This innovative two-stage TVR framework introduces a novel one-to-many coarse-to-fine alignment paradigm, imitating the human cognitive process of identifying specific items within a large collection. Our method employs a Focused-view Fusion Network with a sophisticated cross-attention mechanism, dynamically aligning and comparing features across multiple videos to capture finer nuances and contextual variations. Extensive experiments on six benchmark datasets confirm that TokenBinder substantially outperforms existing state-of-the-art methods. These results demonstrate its robustness and the effectiveness of its fine-grained alignment in bridging intra- and inter-modality information gaps in TVR tasks.
Abstract (translated)
文本-视频检索（TVR）方法通常通过粗粒度、细粒度或结合（粗-细）方式对查询-候选对进行匹配。然而，这些框架主要采用一对一（查询）一对应（候选者）的匹配范式，很难区分候选者之间的细微差异，导致频繁的匹配错误。受到人类认知科学中比较判断的影响，我们提出了TokenBinder。这个创新的两阶段TVR框架引入了一种新颖的一对多粗-细对齐范式，类似于人类认知过程中在大规模集合中识别特定项的过程。我们的方法采用了一个具有复杂交叉注意力的聚焦视图融合网络，动态地对多个视频中的特征进行同步对齐和比较，以捕捉更细粒度和上下文变化。在六个基准数据集上的大量实验证实，TokenBinder显著优于现有最先进的方法。这些结果证明了TokenBinder的稳健性以及其在TVR任务中细粒度对齐的有效性。
URL
https://arxiv.org/abs/2409.19865
