Abstract
Integrating audio and visual data for training multimodal foundational models remains challenging. We present Audio-Video Vector Alignment (AVVA), which aligns audiovisual (AV) scene content beyond mere temporal synchronization via a Large Language Model (LLM)-based data curation pipeline. Specifically, AVVA scores and selects high-quality training clips using Whisper (speech-based audio foundation model) for audio and DINOv2 for video within a dual-encoder contrastive learning framework. Evaluations on AudioCaps, VALOR, and VGGSound demonstrate that this approach can achieve significant accuracy gains with substantially less curated data. For instance, AVVA yields a 7.6% improvement in top-1 accuracy for audio-to-video retrieval on VGGSound compared to ImageBind, despite training on only 192 hours of carefully filtered data (vs. 5800+ hours). Moreover, an ablation study highlights that trading data quantity for data quality improves performance, yielding respective top-3 accuracy increases of 47.8, 48.4, and 58.0 percentage points on AudioCaps, VALOR, and VGGSound over uncurated baselines. While these results underscore AVVA's data efficiency, we also discuss the overhead of LLM-driven curation and how it may be scaled or approximated in larger domains. Overall, AVVA provides a viable path toward more robust, text-free audiovisual learning with improved retrieval accuracy.
Abstract (translated)
将音频和视觉数据整合用于训练多模态基础模型仍然面临挑战。我们提出了一种名为音视频向量对齐（Audio-Video Vector Alignment，AVVA）的方法，该方法通过基于大型语言模型（LLM）的数据整理管道，在超越单纯时间同步的基础上实现了视听场景内容的对齐。具体来说，AVVA利用Whisper（一种基于语音的音频基础模型）处理音频，并使用DINOv2处理视频，在一个双编码器对比学习框架中评分和选择高质量的训练片段。 在AudioCaps、VALOR和VGGSound数据集上的评估表明，这种方法能够在大幅减少整理的数据量的情况下实现显著的准确率提升。例如，在与ImageBind相比时，AVVA仅使用了192小时精心过滤后的数据（相比之下，ImageBind使用的是5800+小时），在VGGSound上实现了音频到视频检索任务7.6%的top-1精度提升。 此外，消融实验表明，以牺牲数据量为代价提高数据质量可以改善性能，在AudioCaps、VALOR和VGGSound上的top-3准确率分别比未经整理的基础线模型提高了47.8%，48.4% 和58.0个百分点。 尽管这些结果强调了AVVA的数据效率，但我们也讨论了由LLM驱动的整理所带来的额外开销以及如何在更大领域内对其进行扩展或近似处理。总体而言，AVVA为实现更为稳健、无需文本引导的视听学习和检索精度提升提供了一条可行的道路。
URL
https://arxiv.org/abs/2503.09205
