Abstract
Audio-Visual Segmentation (AVS) aims to segment sound-producing objects in video frames based on the associated audio signal. Prevailing AVS methods typically adopt an audio-centric Transformer architecture, where object queries are derived from audio features. However, audio-centric Transformers suffer from two limitations: perception ambiguity caused by the mixed nature of audio, and weakened dense prediction ability due to visual detail loss. To address these limitations, we propose a new Vision-Centric Transformer (VCT) framework that leverages vision-derived queries to iteratively fetch corresponding audio and visual information, enabling queries to better distinguish between different sounding objects from mixed audio and accurately delineate their contours. Additionally, we also introduce a Prototype Prompted Query Generation (PPQG) module within our VCT framework to generate vision-derived queries that are both semantically aware and visually rich through audio prototype prompting and pixel context grouping, facilitating audio-visual information aggregation. Extensive experiments demonstrate that our VCT framework achieves new state-of-the-art performances on three subsets of the AVSBench dataset. The code is available at this https URL.
Abstract (translated)
音频视觉分割（AVS）的目标是基于关联的音频信号在视频帧中对发声物体进行分割。现有的AVS方法通常采用以音频为中心的Transformer架构，其中对象查询是从音频特征派生出来的。然而，这类音频中心的Transformer存在两个限制：一是由于音频混合特性导致的感觉模糊问题，二是因为视觉细节损失而导致密集预测能力减弱的问题。 为了解决这些问题，我们提出了一种新的基于视觉的Transformer（VCT）框架，该框架利用来自视觉的信息生成查询，并迭代地获取相应的音频和视觉信息。这使得查询能够更好地从混合音频中区分不同的发声物体，并准确描绘它们的轮廓。 此外，我们在VCT框架内还引入了一个原型提示查询生成（PPQG）模块，通过音频原型提示和像素上下文分组来生成既具有语义感知又能丰富视觉特征的基于视觉的查询。这有助于实现音频与视觉信息的有效聚合。 经过广泛的实验验证，我们的VCT框架在AVSBench数据集的三个子集中实现了新的最先进性能。相关代码可在[此链接]（请将此处的URL替换为实际提供的URL）获取。
URL
https://arxiv.org/abs/2506.23623
