Abstract
Text-video retrieval tasks have seen significant improvements due to the recent development of large-scale vision-language pre-trained models. Traditional methods primarily focus on video representations or cross-modal alignment, while recent works shift toward enriching text expressiveness to better match the rich semantics in videos. However, these methods use only interactions between text and frames/video, and ignore rich interactions among the internal frames within a video, so the final expanded text cannot capture frame contextual information, leading to disparities between text and video. In response, we introduce Energy-Aware Fine-Grained Relationship Learning Network (EagleNet) to generate accurate and context-aware enriched text embeddings. Specifically, the proposed Fine-Grained Relationship Learning mechanism (FRL) first constructs a text-frame graph by the generated text candidates and frames, then learns relationships among texts and frames, which are finally used to aggregate text candidates into an enriched text embedding that incorporates frame contextual information. To further improve fine-grained relationship learning in FRL, we design Energy-Aware Matching (EAM) to model the energy of text-frame interactions and thus accurately capture the distribution of real text-video pairs. Moreover, for more effective cross-modal alignment and stable training, we replace the conventional softmax-based contrastive loss with the sigmoid loss. Extensive experiments have demonstrated the superiority of EagleNet across MSRVTT, DiDeMo, MSVD, and VATEX. Codes are available at this https URL.
Abstract (translated)
文本-视频检索任务因近期大规模视觉-语言预训练模型的发展而取得显著进步。传统方法主要关注视频表示或跨模态对齐，而近期研究转向丰富文本表达以更好匹配视频中的丰富语义。然而，这些方法仅利用文本与帧/视频之间的交互，忽略了视频内部帧之间的丰富关联，导致最终扩展的文本无法捕获帧上下文信息，造成文本与视频之间的不匹配。针对此问题，我们提出能量感知细粒度关系学习网络（EagleNet），以生成准确且具备上下文感知能力的丰富文本嵌入。具体而言，所提出的细粒度关系学习机制（FRL）首先通过生成的文本候选帧与视频帧构建文本-帧图，进而学习文本与帧之间的关系，最终将这些关系用于聚合文本候选，形成融合帧上下文信息的丰富文本嵌入。为进一步提升FRL中的细粒度关系学习，我们设计能量感知匹配（EAM）对文本-帧交互的能量进行建模，从而准确捕获真实文本-视频对的分布。此外，为实现更有效的跨模态对齐与稳定训练，我们将传统的基于softmax的对比损失替换为sigmoid损失。大量实验表明，EagleNet在MSRVTT、DiDeMo、MSVD和VATEX数据集上均具有优越性。代码已公开于此链接。
URL
https://arxiv.org/abs/2603.25267
