Abstract
The explosive growth of video streaming presents challenges in achieving high accuracy and low training costs for video-language retrieval. However, existing methods rely on large-scale pre-training to improve video retrieval performance, resulting in significant computational demands. Additionally, the fine-grained information in videos and texts remains underexplored. To alleviate these problems, we propose a novel framework to learn fine-grained features for better alignment and introduce an inference pipeline to improve performance without additional training. Specifically, we employ coarse-to-fine objectives to understand the semantic information of video-text pairs, including contrastive and matching learning. The fine-grained data used for training is obtained through the Granularity-Aware Representation module, which is designed based on similarity analysis between video frames and words in captions. Furthermore, we observe that the repetition of keywords in the original captions, referred to as "Repetition", can enhance retrieval performance and improve alignment between video and text. Based on this insight, we propose a novel and effective inference pipeline that incorporates a voting mechanism and a new Matching Entropy metric to achieve better retrieval performance without requiring additional pre-training. Experimental results on four benchmarks demonstrate that the proposed method outperforms previous approaches. Additionally, our inference pipeline achieves significant performance improvements, with a 2.1% increase in Recall@1 on the MSR-VTT dataset and a 1.6% increase on the DiDeMo dataset.
Abstract (translated)
视频流媒体的爆炸性增长带来了在实现高精度和低成本训练的同时进行视频-语言检索的挑战。然而，现有的方法依赖大规模预训练来提升视频检索性能，导致了巨大的计算需求。此外，视频和文本中的细粒度信息仍未得到充分探索。为了解决这些问题，我们提出了一种新的框架，用于学习更精细的特征以实现更好的对齐，并引入了一个推理管道，在不增加额外训练的情况下提高性能。 具体来说，我们采用了从粗到细的目标来理解视频-文本对的语义信息，包括对比学习和匹配学习。通过基于视频帧与字幕中单词之间的相似性分析设计的Granularity-Aware Representation模块获得了用于训练的细粒度数据。此外，我们观察到在原始字幕中重复关键词（称为“重复”）可以提升检索性能，并增强视频与文本之间的一致性。基于这一发现，我们提出了一种新的且有效的推理管道，该管道整合了投票机制和一种新的匹配熵指标，在无需额外预训练的情况下实现更好的检索性能。 实验结果表明，所提方法在四个基准测试数据集上超越了先前的方法。此外，我们的推理管道取得了显著的性能提升：在MSR-VTT数据集上，Recall@1提高了2.1%，而在DiDeMo数据集上的提高为1.6%。
URL
https://arxiv.org/abs/2508.14812
