Abstract
Text-to-video retrieval requires precise alignment between language and temporally rich video signals. Existing methods predominantly exploit visual cues and often overlook complementary audio semantics or adopt coarse fusion strategies, leading to suboptimal multimodal representations. We present GAID, a framework that jointly address this gap via two key components: (i) a Frame-level Gated Fusion (FGF) that adaptively integrates audio and visual features under textual guidance, enabling fine-grained temporal alignment; and (ii) a Directional Adaptive Semantic Perturbation (DASP) that injects structure-aware perturbations into text embeddings, enhancing robustness and discrimination without incurring multi-pass inference. These modules complement each other -- fusion reduces modality gaps while perturbation regularizes cross-modal matching -- yielding more stable and expressive representations. Extensive experiments on MSR-VTT, DiDeMo, LSMDC, and VATEX show consistent state-of-the-art results across all retrieval metrics with notable efficiency gains. Our code is available at this https URL.
Abstract (translated)
文本到视频检索要求语言与时间丰富的视频信号之间精确对齐。现有方法主要利用视觉线索，并常常忽视互补的音频语义或采用粗略的融合策略，导致多模态表示次优。我们提出了GAID框架，通过两个关键组件来共同解决这一问题：(i) 帧级门控融合（FGF），在文本引导下自适应地整合音频和视觉特征，从而实现精细的时间对齐；以及(ii) 方向性自适应语义扰动（DASP），将结构感知的扰动注入到文本嵌入中，增强鲁棒性和区分度而不产生多遍推理。这些模块相互补充——融合减少模态差距而扰动则规范跨模态匹配，从而生成更稳定和表达性的表示。在MSR-VTT、DiDeMo、LSMDC 和 VATEX 数据集上的广泛实验显示，在所有检索指标上均取得了一致的最优结果，并且效率显著提高。我们的代码可在提供的链接获取。
URL
https://arxiv.org/abs/2508.01711
