Abstract
In this work, we tackle the problem of text-to-video retrieval (T2VR). Inspired by the success of late interaction techniques in text-document, text-image, and text-video retrieval, our approach, Video-ColBERT, introduces a simple and efficient mechanism for fine-grained similarity assessment between queries and videos. Video-ColBERT is built upon 3 main components: a fine-grained spatial and temporal token-wise interaction, query and visual expansions, and a dual sigmoid loss during training. We find that this interaction and training paradigm leads to strong individual, yet compatible, representations for encoding video content. These representations lead to increases in performance on common text-to-video retrieval benchmarks compared to other bi-encoder methods.
Abstract (translated)
在这项工作中，我们解决了文本到视频检索（T2VR）的问题。受文本文档、文本图像和文本视频检索中后期交互技术成功的启发，我们的方法Video-ColBERT引入了一种简单而高效的机制，用于查询与视频之间的细粒度相似性评估。Video-ColBERT基于三个主要组件构建：精细的空间和时间标记级交互、查询和视觉扩展以及训练期间的双重Sigmoid损失。我们发现这种交互和训练范式导致了编码视频内容的强大且兼容的表示形式。这些表示形式在常见的文本到视频检索基准测试中优于其他双编码器方法，提高了性能。
URL
https://arxiv.org/abs/2503.19009
