Abstract
General audio understanding is a fundamental goal for large audio-language models, with audio captioning serving as a cornerstone task for their development. However, progress in this domain is hindered by existing datasets, which lack the scale and descriptive granularity required to train truly versatile models. To address this gap, we introduce ACAVCaps, a new large-scale, fine-grained, and multi-faceted audio captioning dataset. Derived from the ACAV100M collection, ACAVCaps is constructed using a multi-expert pipeline that analyzes audio from diverse perspectives-including speech, music, and acoustic properties-which are then synthesized into rich, detailed descriptions by a large language model. Experimental results demonstrate that models pre-trained on ACAVCaps exhibit substantially stronger generalization capabilities on various downstream tasks compared to those trained on other leading captioning datasets. The dataset is available at this https URL.
Abstract (translated)
通用音频理解是大型音频-语言模型的基本目标，而音频描述任务则是其发展的基石。然而，该领域的进展受到现有数据集的制约——这些数据集缺乏训练真正通用模型所需的规模和描述粒度。为填补这一空白，我们推出了ACAVCaps，这是一个全新的大规模、细粒度、多角度的音频描述数据集。该数据集源自ACAV100M集合，通过多专家流水线构建：该流水线从语音、音乐及声学特性等多个维度分析音频，再由大语言模型综合生成丰富详尽的描述。实验结果表明，在ACAVCaps上预训练的模型，相较于使用其他领先描述数据集训练的模型，在各类下游任务中展现出显著更强的泛化能力。数据集已通过此链接公开。
URL
https://arxiv.org/abs/2603.24038
