Abstract
Given a text query, partially relevant video retrieval (PRVR) seeks to find untrimmed videos containing pertinent moments in a database. For PRVR, clip modeling is essential to capture the partial relationship between texts and videos. Current PRVR methods adopt scanning-based clip construction to achieve explicit clip modeling, which is information-redundant and requires a large storage overhead. To solve the efficiency problem of PRVR methods, this paper proposes GMMFormer, a \textbf{G}aussian-\textbf{M}ixture-\textbf{M}odel based Trans\textbf{former} which models clip representations implicitly. During frame interactions, we incorporate Gaussian-Mixture-Model constraints to focus each frame on its adjacent frames instead of the whole video. Then generated representations will contain multi-scale clip information, achieving implicit clip modeling. In addition, PRVR methods ignore semantic differences between text queries relevant to the same video, leading to a sparse embedding space. We propose a query diverse loss to distinguish these text queries, making the embedding space more intensive and contain more semantic information. Extensive experiments on three large-scale video datasets (\ie, TVR, ActivityNet Captions, and Charades-STA) demonstrate the superiority and efficiency of GMMFormer.
Abstract (translated)
部分相关视频检索（PRVR）旨在在数据库中寻找未剪辑的视频，其中关键时刻相关。对于PRVR，片段建模对于捕捉文本和视频之间的部分关系至关重要。当前的PRVR方法采用扫描基于片段构建的方法来实现显式的片段建模，这是冗余的信息，需要较大的存储开销。为解决PRVR方法的效率问题，本文提出了GMMFormer，一种基于Transformer的带有Gaussian-Mixture-Model约束的片段建模方法。在帧交互过程中，我们将GMM-Mixture-Model约束融入其中，使每个帧集中于其相邻帧而不是整个视频。这样生成的表示将包含多尺度片段信息，实现隐式片段建模。此外，PRVR方法忽略了相关视频文本查询之间的语义差异，导致稀疏嵌入空间。我们提出了一个查询多样损失，以区分这些文本查询，使嵌入空间更加丰富，包含更多的语义信息。在三个大型视频数据集（即TVR、ActivityNet Captions和Charades-STA）上的大量实验证明GMMFormer的优越性和高效性。
URL
https://arxiv.org/abs/2310.05195
