Abstract
Cinematic Audio Source Separation (CASS) aims to decompose mixed film audio into speech, music, and sound effects, enabling applications like dubbing and remastering. Existing CASS approaches are audio-only, overlooking the inherent audio-visual nature of films, where sounds often align with visual cues. We present the first framework for audio-visual CASS (AV-CASS), leveraging visual context to enhance separation quality. Our method formulates CASS as a conditional generative modeling problem using conditional flow matching, enabling multimodal audio source separation. To address the lack of cinematic datasets with isolated sound tracks, we introduce a training data synthesis pipeline that pairs in-the-wild audio and video streams (e.g., facial videos for speech, scene videos for effects) and design a dedicated visual encoder for this dual-stream setup. Trained entirely on synthetic data, our model generalizes effectively to real-world cinematic content and achieves strong performance on synthetic, real-world, and audio-only CASS benchmarks. Code and demo are available at \url{this https URL}.
Abstract (translated)
电影音频源分离（CASS）旨在将混合的电影音频分解为语音、音乐和音效，以支持配音和重制等应用。现有CASS方法仅依赖音频，忽略了电影固有的音视特性——声音常与视觉线索相关联。我们首次提出了音视联合的CASS框架（AV-CASS），利用视觉上下文提升分离质量。该方法将CASS建模为基于条件流匹配的条件生成问题，实现多模态音频源分离。针对缺乏带独立音轨的电影数据集问题，我们设计了训练数据合成流程，将野外采集的音频-视频流进行配对（如面部视频对应语音，场景视频对应音效），并为此双流架构设计了专用视觉编码器。模型完全基于合成数据训练，能有效泛化至真实电影内容，在合成、真实及纯音频CASS基准测试中均表现优异。代码及演示已公开于\url{此https链接}。
URL
https://arxiv.org/abs/2603.26113
