Abstract
The Text-to-Video Retrieval (T2VR) task aims to retrieve unlabeled videos by textual queries with the same semantic meanings. Recent CLIP-based approaches have explored two frameworks: Two-Tower versus Single-Tower framework, yet the former suffers from low effectiveness, while the latter suffers from low efficiency. In this study, we explore a new Hybrid-Tower framework that can hybridize the advantages of the Two-Tower and Single-Tower framework, achieving high effectiveness and efficiency simultaneously. We propose a novel hybrid method, Fine-grained Pseudo-query Interaction and Generation for T2VR, ie, PIG, which includes a new pseudo-query generator designed to generate a pseudo-query for each video. This enables the video feature and the textual features of pseudo-query to interact in a fine-grained manner, similar to the Single-Tower approaches to hold high effectiveness, even before the real textual query is received. Simultaneously, our method introduces no additional storage or computational overhead compared to the Two-Tower framework during the inference stage, thus maintaining high efficiency. Extensive experiments on five commonly used text-video retrieval benchmarks demonstrate that our method achieves a significant improvement over the baseline, with an increase of $1.6\% \sim 3.9\%$ in R@1. Furthermore, our method matches the efficiency of Two-Tower models while achieving near state-of-the-art performance, highlighting the advantages of the Hybrid-Tower framework.
Abstract (translated)
文本到视频检索（T2VR）任务的目标是通过具有相同语义含义的文本查询来检索未标记的视频。最近基于CLIP的方法探索了两种框架：Two-Tower与Single-Tower框架，前者在有效性方面表现不佳，而后者在效率方面存在不足。在这项研究中，我们提出了一种新的Hybrid-Tower框架，该框架结合了Two-Tower和Single-Tower框架的优点，在保持高效率的同时实现高有效性。我们提出了一种新颖的混合方法——细粒度伪查询交互与生成（PIG），它包括一个新的伪查询生成器，旨在为每个视频生成一个伪查询。这使得视频特征与伪查询的文本特征能够以细粒度方式进行互动，类似于Single-Tower方法，在实际收到文本查询之前就能保持高有效性。同时，我们的方法在推理阶段不引入额外的存储或计算开销，从而保持了高效率。在五个常用的文字-视频检索基准上进行的广泛实验表明，与基线相比，我们的方法取得了显著改进，R@1提升了$1.6\% \sim 3.9\%$。此外，我们的方法达到了Two-Tower模型的效率水平，并实现了接近最先进性能的效果，突显了Hybrid-Tower框架的优势。
URL
https://arxiv.org/abs/2509.04773
