Abstract
Text-Video Retrieval (TVR) aims to align and associate relevant video content with corresponding natural language queries. Most existing TVR methods are based on large-scale pre-trained vision-language models (e.g., CLIP). However, due to the inherent plain structure of CLIP, few TVR methods explore the multi-scale representations which offer richer contextual information for a more thorough understanding. To this end, we propose MUSE, a multi-scale mamba with linear computational complexity for efficient cross-resolution modeling. Specifically, the multi-scale representations are generated by applying a feature pyramid on the last single-scale feature map. Then, we employ the Mamba structure as an efficient multi-scale learner to jointly learn scale-wise representations. Furthermore, we conduct comprehensive studies to investigate different model structures and designs. Extensive results on three popular benchmarks have validated the superiority of MUSE.
Abstract (translated)
文本-视频检索（TVR）旨在将相关视频内容与相应的自然语言查询对齐和关联。大多数现有的TVR方法都是基于大规模预训练的视觉-语言模型（例如，CLIP）。然而，由于CLIP固有的简单结构，很少有TVR方法探索多尺度表示，这些多尺度表示能提供对更全面理解的丰富上下文信息。为此，我们提出了MUSE，一种具有线性计算复杂性的多尺度MAMBA，用于高效的跨分辨率建模。具体来说，多尺度表示是由在最后一个单尺度特征图上应用特征金字塔生成的。然后，我们利用Mamba结构作为高效的多尺度学习器，共同学习尺度上的表示。此外，我们进行了全面的研究，以探究不同的模型结构和设计。在三个流行的基准测试上，MUSE取得了很好的结果，验证了其优越性。
URL
https://arxiv.org/abs/2408.10575
