Abstract
Propelled by the breakthrough in deep generative models, audio-to-image generation has emerged as a pivotal cross-model task that converts complex auditory signals into rich visual representations. However, previous works only focus on single-source audio inputs for image generation, ignoring the multi-source characteristic in natural auditory scenes, thus limiting the performance in generating comprehensive visual content. To bridge this gap, a method called MACS is proposed to conduct multi-source audio-to-image generation. This is the first work that explicitly separates multi-source audio to capture the rich audio components before image generation. MACS is a two-stage method. In the first stage, multi-source audio inputs are separated by a weakly supervised method, where the audio and text labels are semantically aligned by casting into a common space using the large pre-trained CLAP model. We introduce a ranking loss to consider the contextual significance of the separated audio signals. In the second stage, efficient image generation is achieved by mapping the separated audio signals to the generation condition using only a trainable adapter and a MLP layer. We preprocess the LLP dataset as the first full multi-source audio-to-image generation benchmark. The experiments are conducted on multi-source, mixed-source, and single-source audio-to-image generation tasks. The proposed MACS outperforms the current state-of-the-art methods in 17 of the 21 evaluation indexes on all tasks and delivers superior visual quality. The code will be publicly available.
Abstract (translated)
深度生成模型的突破推动了音频到图像生成任务的发展，这一任务将复杂的听觉信号转换为丰富的视觉表示，并在跨模态任务中占据重要地位。然而，之前的研究仅关注单一来源音频输入的图像生成，忽略了自然听觉场景中的多源特性，从而限制了全面视觉内容生成的表现力。为了弥补这一不足，提出了一种名为MACS的方法来实现多源音频到图像的生成。这是第一个明确分离多源音频以捕捉丰富音频成分并在图像生成前进行处理的工作。 MACS是一种两阶段方法。在第一阶段中，使用弱监督方式将多源音频输入进行分离，并通过将音频和文本标签投射到一个共同空间内（利用大型预训练的CLAP模型）来实现语义对齐。我们引入了一个排名损失函数以考虑分离后音频信号中的上下文重要性。在第二阶段中，仅通过可训练适配器和MLP层映射分离后的音频信号到生成条件，从而实现了高效的图像生成。 我们将LLP数据集预处理为首个完整的多源音频到图像生成基准测试数据集，并在多源、混合源及单一来源的音频到图像生成任务上进行了实验。所提出的MACS方法在所有21个评估指标中的17项中超越了当前最先进的方法，同时提供了更优的视觉质量。代码将公开发布。
URL
https://arxiv.org/abs/2503.10287
