Abstract
The success of CLIP has driven substantial progress in text-video retrieval. However, current methods often suffer from "blind" feature interaction, where the model struggles to discern key visual information from background noise due to the sparsity of textual queries. To bridge this gap, we draw inspiration from human cognitive behavior and propose the Human Vision-Driven (HVD) model. Our framework establishes a coarse-to-fine alignment mechanism comprising two key components: the Frame Features Selection Module (FFSM) and the Patch Features Compression Module (PFCM). FFSM mimics the human macro-perception ability by selecting key frames to eliminate temporal redundancy. Subsequently, PFCM simulates micro-perception by aggregating patch features into salient visual entities through an advanced attention mechanism, enabling precise entity-level matching. Extensive experiments on five benchmarks demonstrate that HVD not only captures human-like visual focus but also achieves state-of-the-art performance.
Abstract (translated)
CLIP模型的成功推动了文本视频检索领域的显著进步。然而，当前的方法往往在“盲”特征交互方面存在问题，即由于文本查询的稀疏性，模型难以从背景噪声中区分出关键视觉信息。为了弥补这一差距，我们借鉴了人类的认知行为，并提出了人眼视图驱动（HVD）模型。我们的框架建立了一个由粗到细的对齐机制，包含两个关键组件：帧特征选择模块（FFSM）和补丁特征压缩模块（PFCM）。FFSM通过选择关键帧来消除时间冗余，模拟了人类宏观感知能力。随后，PFCM通过先进的注意力机制聚合补丁特征以形成显著视觉实体，从而模仿微观感知并实现精确的实体级别匹配。 在五个基准测试中的大量实验表明，HVD不仅能够捕捉到类似人的视觉关注点，还实现了最先进的性能。
URL
https://arxiv.org/abs/2601.16155
