Abstract
Partially Relevant Video Retrieval~(PRVR) aims to retrieve a video where a specific segment is relevant to a given text query. Typical training processes of PRVR assume a one-to-one relationship where each text query is relevant to only one video. However, we point out the inherent ambiguity between text and video content based on their conceptual scope and propose a framework that incorporates this ambiguity into the model learning process. Specifically, we propose Ambiguity-Restrained representation Learning~(ARL) to address ambiguous text-video pairs. Initially, ARL detects ambiguous pairs based on two criteria: uncertainty and similarity. Uncertainty represents whether instances include commonly shared context across the dataset, while similarity indicates pair-wise semantic overlap. Then, with the detected ambiguous pairs, our ARL hierarchically learns the semantic relationship via multi-positive contrastive learning and dual triplet margin loss. Additionally, we delve into fine-grained relationships within the video instances. Unlike typical training at the text-video level, where pairwise information is provided, we address the inherent ambiguity within frames of the same untrimmed video, which often contains multiple contexts. This allows us to further enhance learning at the text-frame level. Lastly, we propose cross-model ambiguity detection to mitigate the error propagation that occurs when a single model is employed to detect ambiguous pairs for its training. With all components combined, our proposed method demonstrates its effectiveness in PRVR.
Abstract (translated)
部分相关视频检索（PRVR）的目标是从大量视频中提取出特定段落与给定文本查询相关的视频。传统PRVR的训练过程假设一个一对一的关系，即每个文本查询仅对应一个视频。然而，我们指出了基于概念范围的文本和视频内容之间的固有歧义，并提出了一个将这种歧义纳入模型学习过程的框架。具体来说，我们提出了一种名为“歧义受限表示学习”（Ambiguity-Restrained Learning, ARL）的方法来解决含有模糊配对的文本-视频问题。 最初，ARL基于两个标准检测模糊配对：不确定性以及相似度。不确定性代表实例是否包含了跨整个数据集常见的上下文信息；而相似度则表明成对的语义重叠。然后，在检测到模糊配对之后，我们的ARL通过多正对比学习和双三元组边缘损失来分层学习语义关系。 此外，我们深入探讨了视频片段内的细粒度关联性。不同于典型的文本-视频层级训练中提供的两两信息，我们针对同一未修剪视频中的帧内固有的歧义进行了处理，通常这些视频包含多个上下文场景。这使我们在文本-帧层次上进一步增强了学习能力。 最后，我们提出了跨模型的歧义检测来缓解当单一模型被用于模糊配对训练时出现的错误传播问题。结合所有组件，我们的方法在PRVR中表现出显著的有效性。
URL
https://arxiv.org/abs/2506.07471
