Abstract
Current text-video retrieval methods mainly rely on cross-modal matching between queries and videos to calculate their similarity scores, which are then sorted to obtain retrieval results. This method considers the matching between each candidate video and the query, but it incurs a significant time cost and will increase notably with the increase of candidates. Generative models are common in natural language processing and computer vision, and have been successfully applied in document retrieval, but their application in multimodal retrieval remains unexplored. To enhance retrieval efficiency, in this paper, we introduce a model-based video indexer named T2VIndexer, which is a sequence-to-sequence generative model directly generating video identifiers and retrieving candidate videos with constant time complexity. T2VIndexer aims to reduce retrieval time while maintaining high accuracy. To achieve this goal, we propose video identifier encoding and query-identifier augmentation approaches to represent videos as short sequences while preserving their semantic information. Our method consistently enhances the retrieval efficiency of current state-of-the-art models on four standard datasets. It enables baselines with only 30\%-50\% of the original retrieval time to achieve better retrieval performance on MSR-VTT (+1.0%), MSVD (+1.8%), ActivityNet (+1.5%), and DiDeMo (+0.2%). The code is available at this https URL.
Abstract (translated)
当前的文本-视频检索方法主要依赖于查询和视频之间的跨模态匹配来计算它们的相似度分数，然后对结果进行排序。这种方法考虑了每个候选视频与查询之间的匹配，但它需要显著的时间成本，并且在候选视频增加时会增加显著。在自然语言处理和计算机视觉中，生成模型很常见，并且在文档检索中取得了成功应用，但在多模态检索中的应用仍然是一个未探索的领域。为了提高检索效率，本文提出了一种基于模型的视频索引器T2VIndexer，它是一种直接生成视频标识的序列到序列生成模型，用于检索具有恒定时间复杂度的候选视频。T2VIndexer的目标是在保持高准确性的同时减少检索时间。为实现这一目标，我们提出了视频标识编码和查询标识增强方法，将视频表示为短序列，同时保留其语义信息。我们的方法在四个标准数据集上 consistent地增强了当前最先进模型的检索效率。它使得仅使用原始检索时间的30\%-50\%的基线模型就能实现更好的 MSR-VTT (+1.0%)、MSVD (+1.8%)、ActivityNet (+1.5%) 和 DiDeMo (+0.2%) 上的检索性能。代码可在此处访问：https://www.xxx.com/。
URL
https://arxiv.org/abs/2408.11432
