Abstract
Content creators often use music to enhance their videos, from soundtracks in movies to background music in video blogs and social media content. However, identifying the best music for a video can be a difficult and time-consuming task. To address this challenge, we propose a novel framework for automatically retrieving a matching music clip for a given video, and vice versa. Our approach leverages annotated music labels, as well as the inherent artistic correspondence between visual and music elements. Distinct from previous cross-modal music retrieval works, our method combines both self-supervised and supervised training objectives. We use self-supervised and label-supervised contrastive learning to train a joint embedding space between music and video. We show the effectiveness of our approach by using music genre labels for the supervised training component, and our framework can be generalized to other music annotations (e.g., emotion, instrument, etc.). Furthermore, our method enables fine-grained control over how much the retrieval process focuses on self-supervised vs. label information at inference time. We evaluate the learned embeddings through a variety of video-to-music and music-to-video retrieval tasks. Our experiments show that the proposed approach successfully combines self-supervised and supervised objectives and is effective for controllable music-video retrieval.
Abstract (translated)
内容创作者经常使用音乐来增强他们的视频，从电影中的配乐到视频博客和社交媒体内容的背景音乐。然而，为视频找到最合适的音乐可能是一项既困难又耗时的任务。为了应对这一挑战，我们提出了一种新型框架，可以自动检索与给定视频相匹配的音乐片段，并且反之亦然。我们的方法利用了带注释的音乐标签，以及视觉元素和音乐元素之间固有的艺术对应关系。与之前的跨模态音乐检索工作不同，我们的方法结合了自我监督和监督学习目标。我们使用自我监督对比学习和标签监督对比学习来训练音乐和视频之间的联合嵌入空间。通过在监督训练组件中使用音乐流派标签，展示了我们方法的有效性，并且我们的框架可以推广到其他类型的音乐注释（如情感、乐器等）。此外，我们的方法能够对检索过程如何在推理时平衡自我监督与标签信息的比例进行细粒度控制。我们通过一系列视频到音乐和音乐到视频的检索任务评估了学习到的嵌入效果。实验结果表明，所提出的方法成功地结合了自我监督和监督目标，并且对于可控制的音乐-视频检索是有效的。
URL
https://arxiv.org/abs/2412.05831
