Abstract
In the field of audio-visual learning, most research tasks focus exclusively on short videos. This paper focuses on the more practical Dense Audio-Visual Event Localization (DAVEL) task, advancing audio-visual scene understanding for longer, {untrimmed} videos. This task seeks to identify and temporally pinpoint all events simultaneously occurring in both audio and visual streams. Typically, each video encompasses dense events of multiple classes, which may overlap on the timeline, each exhibiting varied durations. Given these challenges, effectively exploiting the audio-visual relations and the temporal features encoded at various granularities becomes crucial. To address these challenges, we introduce a novel \ul{CC}Net, comprising two core modules: the Cross-Modal Consistency \ul{C}ollaboration (CMCC) and the Multi-Temporal Granularity \ul{C}ollaboration (MTGC). Specifically, the CMCC module contains two branches: a cross-modal interaction branch and a temporal consistency-gated branch. The former branch facilitates the aggregation of consistent event semantics across modalities through the encoding of audio-visual relations, while the latter branch guides one modality's focus to pivotal event-relevant temporal areas as discerned in the other modality. The MTGC module includes a coarse-to-fine collaboration block and a fine-to-coarse collaboration block, providing bidirectional support among coarse- and fine-grained temporal features. Extensive experiments on the UnAV-100 dataset validate our module design, resulting in a new state-of-the-art performance in dense audio-visual event localization. The code is available at \url{this https URL}.
Abstract (translated)
在视听学习领域，大多数研究任务仅关注短片。本文聚焦于更为实用的密集视听事件定位（DAVEL）任务，推动了对更长、未剪辑视频的视听场景理解的发展。该任务旨在同时识别并精确定位音频和视频流中所有同时发生的事件。通常情况下，每个视频包含多个类别的密集事件，并且这些事件可能在时间线上重叠，各自持续的时间也不同。鉴于这些挑战，有效地利用跨模态关系以及以各种粒度编码的时间特征变得至关重要。为了解决这些问题，我们引入了一种新的\ul{CC}Net，它包括两个核心模块：跨模态一致性协作（CMCC）和多时间粒度协作（MTGC）。具体而言，CMCC模块包含两条分支：一条跨模态交互分支和一条时间一致性门控分支。前一支路通过编码视听关系促进跨模态一致事件语义的聚合，而后一支路引导一种模式聚焦于在另一种模式中识别到的关键事件相关时间区域。MTGC模块包括一个粗至细协作块和一个细至粗协作块，为粗粒度和细粒度的时间特征提供双向支持。我们在UnAV-100数据集上的广泛实验验证了我们的模块设计的有效性，并取得了密集视听事件定位的新最佳性能。代码可在\url{此 https URL}获取。
URL
https://arxiv.org/abs/2412.12628
