Abstract
Video has emerged as a favored multimedia format on the internet. To better gain video contents, a new topic HIREST is presented, including video retrieval, moment retrieval, moment segmentation, and step-captioning. The pioneering work chooses the pre-trained CLIP-based model for video retrieval, and leverages it as a feature extractor for other three challenging tasks solved in a multi-task learning paradigm. Nevertheless, this work struggles to learn the comprehensive cognition of user-preferred content, due to disregarding the hierarchies and association relations across modalities. In this paper, guided by the shallow-to-deep principle, we propose a query-centric audio-visual cognition (QUAG) network to construct a reliable multi-modal representation for moment retrieval, segmentation and step-captioning. Specifically, we first design the modality-synergistic perception to obtain rich audio-visual content, by modeling global contrastive alignment and local fine-grained interaction between visual and audio modalities. Then, we devise the query-centric cognition that uses the deep-level query to perform the temporal-channel filtration on the shallow-level audio-visual representation. This can cognize user-preferred content and thus attain a query-centric audio-visual representation for three tasks. Extensive experiments show QUAG achieves the SOTA results on HIREST. Further, we test QUAG on the query-based video summarization task and verify its good generalization.
Abstract (translated)
视频已作为互联网上受欢迎的多媒体格式出现。为了更好地获取视频内容，提出了一个新的主题HIREST，包括视频检索、关键时刻检索、关键时刻分割和步骤注释。这项开创性工作选择了基于预训练CLIP模型的视频检索，并将其用作多任务学习范式中其他三个具有挑战性的任务的特征提取器。然而，由于忽略了跨模态之间的层次关系和关联关系，这项工作难以学习用户偏好内容的全面认知。在本文中，遵循浅层到深层的原则，我们提出了一种以查询为中心的视听认知（QUAG）网络来构建关键时刻检索、分割和步骤注释的可靠多模态表示。具体来说，首先设计了协同感知模式来获取丰富的视听内容，通过建模视觉和音频模态之间的全局对比对齐和局部精细交互。然后，我们制定了以查询为中心的认知方法，使用深层查询在浅层视听表示上执行时间通道过滤。这可以识别用户偏好内容，并为三个任务获得以查询为中心的视听表示。广泛的实验表明QUAG在HIREST中达到了最先进的结果。进一步，我们在基于查询的视频摘要任务上测试了QUAG并验证了其良好的泛化能力。
URL
https://arxiv.org/abs/2412.13543
