Abstract
In a retrieval system, simultaneously achieving search accuracy and efficiency is inherently challenging. This challenge is particularly pronounced in partially relevant video retrieval (PRVR), where incorporating more diverse context representations at varying temporal scales for each video enhances accuracy but increases computational and memory costs. To address this dichotomy, we propose a prototypical PRVR framework that encodes diverse contexts within a video into a fixed number of prototypes. We then introduce several strategies to enhance text association and video understanding within the prototypes, along with an orthogonal objective to ensure that the prototypes capture a diverse range of content. To keep the prototypes searchable via text queries while accurately encoding video contexts, we implement cross- and uni-modal reconstruction tasks. The cross-modal reconstruction task aligns the prototypes with textual features within a shared space, while the uni-modal reconstruction task preserves all video contexts during encoding. Additionally, we employ a video mixing technique to provide weak guidance to further align prototypes and associated textual representations. Extensive evaluations on TVR, ActivityNet-Captions, and QVHighlights validate the effectiveness of our approach without sacrificing efficiency.
Abstract (translated)
在检索系统中，同时实现搜索准确性和效率是非常具有挑战性的。这一问题尤其体现在部分相关视频检索（PRVR）中，在此场景下，为了提升准确性而加入更多样化的上下文表示会随时间尺度变化增加计算和内存成本。为了解决这一矛盾，我们提出了一种原型式的PRVR框架，该框架将视频中的多样化背景信息编码为固定数量的原型。接下来，我们引入了几种策略来增强文本关联及对原型中视频的理解，并设置了一个正交目标以确保这些原型能够捕捉到内容的多样性。为了在通过文字查询搜索时保持原型可查找性的同时准确地编码视频上下文，我们实施了跨模态和单模态重构任务。其中，跨模态重构任务将原型与文本特征对齐于共享空间中，而单模态重构任务则保留所有视频背景信息的完整性。此外，我们还采用了一种视频混合技术来提供弱引导以进一步对齐原型与其相关的文本表示。我们在TVR、ActivityNet-Captions和QVHighlights数据集上的广泛评估验证了我们的方法的有效性，并且在效率方面没有牺牲性能。
URL
https://arxiv.org/abs/2504.13035
