Abstract
In the rapidly expanding domain of web video content, the task of text-video retrieval has become increasingly critical, bridging the semantic gap between textual queries and video data. This paper introduces a novel data-centric approach, Generalized Query Expansion (GQE), to address the inherent information imbalance between text and video, enhancing the effectiveness of text-video retrieval systems. Unlike traditional model-centric methods that focus on designing intricate cross-modal interaction mechanisms, GQE aims to expand the text queries associated with videos both during training and testing phases. By adaptively segmenting videos into short clips and employing zero-shot captioning, GQE enriches the training dataset with comprehensive scene descriptions, effectively bridging the data imbalance gap. Furthermore, during retrieval, GQE utilizes Large Language Models (LLM) to generate a diverse set of queries and a query selection module to filter these queries based on relevance and diversity, thus optimizing retrieval performance while reducing computational overhead. Our contributions include a detailed examination of the information imbalance challenge, a novel approach to query expansion in video-text datasets, and the introduction of a query selection strategy that enhances retrieval accuracy without increasing computational costs. GQE achieves state-of-the-art performance on several benchmarks, including MSR-VTT, MSVD, LSMDC, and VATEX, demonstrating the effectiveness of addressing text-video retrieval from a data-centric perspective.
Abstract (translated)
在快速扩展的 Web 视频内容领域，文本视频检索任务变得越来越关键，将文本查询和视频数据之间的语义差距缩小。本文提出了一种新颖的数据中心方法，通用查询扩展（GQE），来解决文本和视频之间的固有信息不平衡，提高文本视频检索系统的有效性。与传统的模型中心方法不同，GQE 旨在在训练和测试阶段扩大与视频相关的文本查询。通过自适应地将视频剪辑为短片段并使用零 shot 字幕，GQE 丰富了训练数据集，有效地弥合了数据不平衡。此外，在检索过程中，GQE 利用大型语言模型（LLM）生成一系列问题，并采用查询选择模块根据相关性和多样性过滤这些问题，从而在减少计算开销的同时优化检索性能。我们的贡献包括对信息不平衡挑战的详细分析、视频文本数据集中的查询扩展新方法和引入查询选择策略以提高检索准确性的详细说明。GQE 在多个基准测试中都取得了最先进的性能，包括 MSR-VTT、MSVD、LSMDC 和 VATEX，这表明从数据中心的角度解决文本视频检索是有效的。
URL
https://arxiv.org/abs/2408.07249
