Abstract
Video question grounding (VideoQG) requires models to answer the questions and simultaneously infer the relevant video segments to support the answers. However, existing VideoQG methods usually suffer from spurious cross-modal correlations, leading to a failure to identify the dominant visual scenes that align with the intended question. Moreover, vision-language models exhibit unfaithful generalization performance and lack robustness on challenging downstream tasks such as VideoQG. In this work, we propose a novel VideoQG framework named Cross-modal Causal Relation Alignment (CRA), to eliminate spurious correlations and improve the causal consistency between question-answering and video temporal grounding. Our CRA involves three essential components: i) Gaussian Smoothing Grounding (GSG) module for estimating the time interval via cross-modal attention, which is de-noised by an adaptive Gaussian filter, ii) Cross-Modal Alignment (CMA) enhances the performance of weakly supervised VideoQG by leveraging bidirectional contrastive learning between estimated video segments and QA features, iii) Explicit Causal Intervention (ECI) module for multimodal deconfounding, which involves front-door intervention for vision and back-door intervention for language. Extensive experiments on two VideoQG datasets demonstrate the superiority of our CRA in discovering visually grounded content and achieving robust question reasoning. Codes are available at this https URL.
Abstract (translated)
视频问题定位（Video Question Grounding，简称VideoQG）要求模型能够回答问题并同时推断出支持答案的相关视频片段。然而，现有的VideoQG方法通常会遇到虚假跨模态相关性的问题，导致难以识别与预期问题一致的主要视觉场景。此外，视觉-语言模型在诸如VideoQG等具有挑战性的下游任务中表现出泛化性能不佳且不够稳健的特点。 针对这些问题，在这项工作中我们提出了一种名为“跨模态因果关系对齐”（Cross-modal Causal Relation Alignment, CRA）的新框架，旨在消除虚假相关性，并提高问题回答与视频时间定位之间的因果一致性。CRA包括三个核心组件： 1. **高斯平滑接地(Gaussian Smoothing Grounding, GSG)模块**：通过跨模态注意力机制估计时间段，并利用自适应的高斯滤波器进行去噪。 2. **跨模态对齐(Cross-Modal Alignment, CMA)**：通过视频片段和问答特征之间的双向对比学习，增强弱监督VideoQG的表现。 3. **明确因果干预(Explicit Causal Intervention, ECI)模块**：用于多模态的反混淆处理，包括视觉的前门干预以及语言的后门干预。 在两个VideoQG数据集上的广泛实验表明，我们的CRA框架在发现视觉接地内容和实现稳健的问题推理方面具有显著优势。相关代码可以在提供的URL地址获取。
URL
https://arxiv.org/abs/2503.07635
