Abstract
Given a text query, partially relevant video retrieval (PRVR) aims to retrieve untrimmed videos containing relevant moments. Due to the lack of moment annotations, the uncertainty lying in clip modeling and text-clip correspondence leads to major challenges. Despite the great progress, existing solutions either sacrifice efficiency or efficacy to capture varying and uncertain video moments. What's worse, few methods have paid attention to the text-clip matching pattern under such uncertainty, exposing the risk of semantic collapse. To address these issues, we present GMMFormer v2, an uncertainty-aware framework for PRVR. For clip modeling, we improve a strong baseline GMMFormer with a novel temporal consolidation module upon multi-scale contextual features, which maintains efficiency and improves the perception for varying moments. To achieve uncertainty-aware text-clip matching, we upgrade the query diverse loss in GMMFormer to facilitate fine-grained uniformity and propose a novel optimal matching loss for fine-grained text-clip alignment. Their collaboration alleviates the semantic collapse phenomenon and neatly promotes accurate correspondence between texts and moments. We conduct extensive experiments and ablation studies on three PRVR benchmarks, demonstrating remarkable improvement of GMMFormer v2 compared to the past SOTA competitor and the versatility of uncertainty-aware text-clip matching for PRVR. Code is available at \url{this https URL}.
Abstract (translated)
给定文本查询，部分相关视频检索（PRVR）旨在检索包含相关时刻的未剪辑视频。由于缺乏时刻注释，剪辑建模和文本-片段匹配带来的不确定性导致了重大挑战。尽管取得了很大进展，但现有的解决方案要么牺牲效率要么牺牲效果来捕捉变化和不确定的视频时刻。更糟糕的是，很少有方法关注在不确定情况下文本片段匹配模式，这揭示了语义崩塌的风险。为了应对这些问题，我们提出了GMMFormer v2，一个 uncertainty-aware 的 PRVR 框架。对于剪辑建模，我们在多尺度上下文特征的基础上，通过一种新颖的时间整合模块来改进一个强大的基线 GMMFormer，从而保持效率并提高对不同时刻的感知。为了实现 uncertainty-aware 的文本片段匹配，我们升级了 GMMFormer 的查询多样损失，以促进细粒度均匀性和提出一种新颖的细粒度文本片段对齐最优损失。它们的协同作用减轻了语义崩塌现象，并生动地促进了文本和时刻之间的准确匹配。我们在三个 PRVR 基准上进行了广泛的实验和消融研究，证明了 GMMFormer v2 相对于过去的 SOTA 竞争者取得了显著的改进，并且 uncertainty-aware 文本片段匹配为 PRVR 带来了丰富的多样性。代码可在此处访问：\url{这个链接}。
URL
https://arxiv.org/abs/2405.13824
