Abstract
While multimodal systems have achieved impressive advances, they typically rely on text-aligned representations rather than directly integrating audio and visual inputs. This reliance can limit the use of acoustic information in tasks requiring nuanced audio understanding. In response, SoundCLIP explores direct audio-visual integration within multimodal large language models (MLLMs) by substituting CLIP's visual tokens with audio representations and selecting sound-relevant patch tokens in models such as LLaVA. We investigate two configurations: (1) projecting audio features into CLIP's visual manifold via a multilayer perceptron trained with InfoNCE on paired audio-video segments, and (2) preserving raw audio embeddings with minimal dimensional adjustments. Experiments with five state-of-the-art audio encoders reveal a fundamental trade-off. While audio-to-video retrieval performance increases dramatically (up to 44 percentage points in Top-1 accuracy) when audio is projected into CLIP's space, text generation quality declines. Encoders pre-trained with text supervision (CLAP, Whisper, ImageBind) maintain stronger generative capabilities than those focused primarily on audiovisual alignment (Wav2CLIP, AudioCLIP), highlighting the value of language exposure for generation tasks. We introduce WhisperCLIP, an architecture that fuses intermediate representations from Whisper, as well as AudioVisual Event Evaluation (AVE-2), a dataset of 580,147 three-second audiovisual clips with fine-grained alignment annotations. Our findings challenge the assumption that stronger cross-modal alignment necessarily benefits all multimodal tasks; instead, a Pareto frontier emerges wherein optimal performance depends on balancing retrieval accuracy with text generation quality. Codes and datasets: this https URL.
Abstract (translated)
尽管多模态系统已经取得了令人印象深刻的进展，但它们通常依赖于文本对齐的表示方式，而不是直接整合音频和视觉输入。这种依赖性可能会限制在需要细腻音频理解的任务中利用声学信息的能力。为此，SoundCLIP 探索了在多模态大型语言模型（MLLMs）中进行直接音视频集成的方法，通过将 CLIP 的视觉标记替换为音频表示，并在如 LLaVA 等模型中选择与声音相关的补丁标记。我们研究了两种配置：(1) 通过 InfoNCE 训练的多层感知机，将音频特征投影到 CLIP 的视觉流形空间；(2) 在保留原始音频嵌入的同时进行最少的维度调整。 使用五种最先进的音频编码器进行实验揭示了一个基本的权衡。当音频被投射进入 CLIP 空间时，音视频检索性能显著提高（Top-1 准确率最多提高了 44 个百分点），但文本生成质量下降了。经过文本监督预训练的编码器（如 CLAP、Whisper 和 ImageBind）比主要关注视听对齐的编码器（Wav2CLIP、AudioCLIP）具有更强的生成能力，这凸显了语言曝光对于生成任务的价值。 我们介绍了 WhisperCLIP 架构，该架构融合了来自 Whisper 的中间表示，并且引入了音频视频事件评估数据集 AVE-2，这是一个包含 580,147 条三秒音视频片段的数据集，每条片段都有精细的对齐注释。我们的研究结果挑战了更强跨模态对齐必然有益于所有多模态任务这一假设；相反地，一个帕累托前沿图出现了，在该图中，最佳性能取决于检索准确性和文本生成质量之间的平衡。 代码和数据集：[这个链接](https://this-url.com)。
URL
https://arxiv.org/abs/2506.10416
