Abstract
Composed Video Retrieval (CoVR) retrieves a target video given a query video and a modification text describing the intended change. Existing CoVR benchmarks emphasize appearance shifts or coarse event changes and therefore do not test the ability to capture subtle, fast-paced temporal differences. We introduce TF-CoVR, the first large-scale benchmark dedicated to temporally fine-grained CoVR. TF-CoVR focuses on gymnastics and diving and provides 180K triplets drawn from FineGym and FineDiving. Previous CoVR benchmarks focusing on temporal aspect, link each query to a single target segment taken from the same video, limiting practical usefulness. In TF-CoVR, we instead construct each <query, modification> pair by prompting an LLM with the label differences between clips drawn from different videos; every pair is thus associated with multiple valid target videos (3.9 on average), reflecting real-world tasks such as sports-highlight generation. To model these temporal dynamics we propose TF-CoVR-Base, a concise two-stage training framework: (i) pre-train a video encoder on fine-grained action classification to obtain temporally discriminative embeddings; (ii) align the composed query with candidate videos using contrastive learning. We conduct the first comprehensive study of image, video, and general multimodal embedding (GME) models on temporally fine-grained composed retrieval in both zero-shot and fine-tuning regimes. On TF-CoVR, TF-CoVR-Base improves zero-shot mAP@50 from 5.92 (LanguageBind) to 7.51, and after fine-tuning raises the state-of-the-art from 19.83 to 25.82.
Abstract (translated)
构成视频检索（CoVR）是指在给定查询视频和描述预期变化的修改文本的情况下，检索目标视频。现有的CoVR基准测试主要侧重于外观的变化或粗略事件的更改，因此无法检测捕捉细微、快速的时间差异的能力。我们引入了TF-CoVR，这是首个专注于时间细粒度构成视频检索的大规模基准测试。TF-CoVR聚焦于体操和跳水项目，并提供了180K个三元组，这些数据来源于FineGym和FineDiving。 之前的侧重于时间方面的CoVR基准测试将每个查询与同一视频中的单个目标片段相连，限制了其实用性。在TF-CoVR中，我们通过提示LLM使用从不同视频剪辑之间提取的标签差异来构建每个<query, modification>对；因此，每一对都关联到多个有效的目标视频（平均为3.9），反映了诸如体育精彩集锦生成等现实世界任务。 为了模拟这些时间动态变化，我们提出了TF-CoVR-Base，这是一种简洁的两阶段训练框架：(i) 在细粒度动作分类上对视频编码器进行预训练以获得具有时间区分性的嵌入；(ii) 使用对比学习方法将构成查询与候选视频对齐。我们在零样本和微调设置下，进行了首个关于图像、视频以及通用多模态嵌入（GME）模型在时间细粒度构成检索中的全面研究。 在TF-CoVR上，TF-CoVR-Base使零样本mAP@50从5.92（LanguageBind）提升到了7.51，在微调后将最先进的水平从19.83提高到了25.82。
URL
https://arxiv.org/abs/2506.05274
