Abstract
Audio-Visual Segmentation (AVS) aims to identify and segment sound-producing objects in videos by leveraging both visual and audio modalities. It has emerged as a significant research area in multimodal perception, enabling fine-grained object-level understanding. In this survey, we present a comprehensive overview of the AVS field, covering its problem formulation, benchmark datasets, evaluation metrics, and the progression of methodologies. We analyze a wide range of approaches, including architectures for unimodal and multimodal encoding, key strategies for audio-visual fusion, and various decoder designs. Furthermore, we examine major training paradigms, from fully supervised learning to weakly supervised and training-free methods. Notably, we provide an extensive comparison of AVS methods across standard benchmarks, highlighting the impact of different architectural choices, fusion strategies, and training paradigms on performance. Finally, we outline the current challenges, such as limited temporal modeling, modality bias toward vision, lack of robustness in complex environments, and high computational demands, and propose promising future directions, including improving temporal reasoning and multimodal fusion, leveraging foundation models for better generalization and few-shot learning, reducing reliance on labeled data through selfand weakly supervised learning, and incorporating higher-level reasoning for more intelligent AVS systems.
Abstract (translated)
音频-视觉分割（AVS）旨在通过利用视觉和听觉两种模式来识别并划分视频中产生声音的对象。它已经成为多模态感知领域的一个重要研究方向，能够实现对物体层面的精细理解。在这篇综述中，我们全面概述了AVS领域的现状，涵盖其问题定义、基准数据集、评估指标以及方法论的发展历程。本文分析了一系列广泛的方法，包括单模和跨模编码架构设计，音频-视觉融合的关键策略，以及各种解码器的设计方案。此外，本文还考察了主要的训练范式，从完全监督学习到弱监督及无训练方法。尤为重要的是，我们对标准基准上的AVS方法进行了详尽比较，突出了不同架构选择、融合策略和训练范式对性能的影响。 最后，本文概述了当前面临的主要挑战，如有限的时间建模能力、偏向视觉的模式偏差、复杂环境下的鲁棒性不足以及计算需求高。同时提出了有前景的发展方向，包括提升时间推理与多模态融合的能力，利用基础模型以提高泛化能力和少量样本学习能力，通过自我监督及弱监督学习减少对标注数据的依赖，并引入高级别推理以构建更为智能的AVS系统。
URL
https://arxiv.org/abs/2508.03724
