Abstract
Precise video retrieval requires multi-modal correlations to handle unseen vocabulary and scenes, becoming more complex for lengthy videos where models must perform effectively without prior training on a specific dataset. We introduce a unified framework that combines a visual matching stream and an aural matching stream with a unique subtitles-based video segmentation approach. Additionally, the aural stream includes a complementary audio-based two-stage retrieval mechanism that enhances performance on long-duration videos. Considering the complex nature of retrieval from lengthy videos and its corresponding evaluation, we introduce a new retrieval evaluation method specifically designed for long-video retrieval to support further research. We conducted experiments on the YouCook2 benchmark, showing promising retrieval performance.
Abstract (translated)
精确的视频检索需要多模态关联来处理未见过的词汇和场景，在长视频中，模型必须在没有特定数据集预训练的情况下有效工作，这使得问题变得更加复杂。我们提出了一种统一框架，结合了视觉匹配流与基于字幕的视频分割方法以及听觉匹配流。此外，听觉流还包括一种增强长时间视频检索性能的音频基两阶段检索机制。鉴于从长视频中检索信息及其评估的复杂性，我们引入了一种专门针对长视频检索设计的新评价方法，以支持进一步的研究。我们在YouCook2基准上进行了实验，展示了有前景的检索性能。
URL
https://arxiv.org/abs/2504.04572
