Abstract
In this paper our objectives are, first, networks that can embed audio and visual inputs into a common space that is suitable for cross-modal retrieval; and second, a network that can localize the object that sounds in an image, given the audio signal. We achieve both these objectives by training from unlabelled video using only audio-visual correspondence (AVC) as the objective function. This is a form of cross-modal self-supervision from video. To this end, we design new network architectures that can be trained for cross-modal retrieval and localizing the sound source in an image, by using the AVC task. We make the following contributions: (i) show that audio and visual embeddings can be learnt that enable both within-mode (e.g. audio-to-audio) and between-mode retrieval; (ii) explore various architectures for the AVC task, including those for the visual stream that ingest a single image, or multiple images, or a single image and multi-frame optical flow; (iii) show that the semantic object that sounds within an image can be localized (using only the sound, no motion or flow information); and (iv) give a cautionary tale on how to avoid undesirable shortcuts in the data preparation.
Abstract (translated)
在本文中，我们的目标首先是可以将音频和视觉输入嵌入到适合跨模态检索的公共空间的网络;第二，给定音频信号，可以定位图像中声音的对象的网络。我们通过仅使用视听对应（AVC）作为目标函数的未标记视频进行训练来实现这两个目标。这是视频中的跨模式自我监督的一种形式。 为此，我们设计了新的网络架构，可以通过使用AVC任务来训练跨模态检索和本地化图像中的声源。我们做出以下贡献：（i）表明可以学习音频和视觉嵌入，以实现内部模式（例如音频到音频）和模式间检索; （ii）探索AVC任务的各种架构，包括摄取单个图像的可视流，或多个图像，或单个图像和多帧光流; （iii）表明在图像中发声的语义对象可以被定位（仅使用声音，没有运动或流动信息）; （iv）就如何避免数据准备中的不良捷径提供警示。
URL
https://arxiv.org/abs/1712.06651
