Abstract
Audio-Visual Video Parsing (AVVP) task aims to identify event categories and their occurrence times in a given video with weakly supervised labels. Existing methods typically fall into two categories: (i) designing enhanced architectures based on attention mechanism for better temporal modeling, and (ii) generating richer pseudo-labels to compensate for the absence of frame-level annotations. However, the first type methods treat noisy segment-level pseudo labels as reliable supervision and the second type methods let indiscriminate attention spread them across all frames, the initial errors are repeatedly amplified during training. To address this issue, we propose a method that combines the Bi-Directional Text Fusion (BiT) module and Category-Aware Temporal Graph (CATS) module. Specifically, we integrate the strengths and complementarity of the two previous research directions. We first perform semantic injection and dynamic calibration on audio and visual modality features through the BiT module, to locate and purify cleaner and richer semantic cues. Then, we leverage the CATS module for semantic propagation and connection to enable precise semantic information dissemination across time. Experimental results demonstrate that our proposed method achieves state-of-the-art (SOTA) performance in multiple key indicators on two benchmark datasets, LLP and UnAV-100.
Abstract (translated)
音频视觉视频解析（Audio-Visual Video Parsing，AVVP）任务旨在通过弱监督标签识别给定视频中的事件类别及其发生时间。现有方法通常分为两类：一类是设计基于注意力机制的增强架构以实现更好的时间建模；另一类是生成更丰富的伪标签来弥补帧级别注释缺失的问题。然而，第一类方法将嘈杂的片段级伪标签视为可靠监督信号，并且第二类方法让无差别注意扩散到所有帧中，在训练过程中初始错误会反复被放大。 为了解决这个问题，我们提出了一种结合双向文本融合（Bi-Directional Text Fusion，BiT）模块和类别感知时间图（Category-Aware Temporal Graph，CATS）模块的方法。具体来说，我们将之前两个研究方向的优势与互补性结合起来。首先，通过BiT模块对音频和视觉模态特征进行语义注入和动态校准，以定位并净化更清洁、更丰富的语义线索。然后，利用CATS模块实现语义传播和连接功能，使精确的语义信息能够在时间维度上传播。 实验结果表明，在LLP和UnAV-100两个基准数据集上的多个关键指标上，我们提出的方法取得了最先进的（State-of-the-Art，SOTA）性能。
URL
https://arxiv.org/abs/2509.04086
