Abstract
We present MGE-LDM, a unified latent diffusion framework for simultaneous music generation, source imputation, and query-driven source separation. Unlike prior approaches constrained to fixed instrument classes, MGE-LDM learns a joint distribution over full mixtures, submixtures, and individual stems within a single compact latent diffusion model. At inference, MGE-LDM enables (1) complete mixture generation, (2) partial generation (i.e., source imputation), and (3) text-conditioned extraction of arbitrary sources. By formulating both separation and imputation as conditional inpainting tasks in the latent space, our approach supports flexible, class-agnostic manipulation of arbitrary instrument sources. Notably, MGE-LDM can be trained jointly across heterogeneous multi-track datasets (e.g., Slakh2100, MUSDB18, MoisesDB) without relying on predefined instrument categories. Audio samples are available at our project page: this https URL.
Abstract (translated)
我们介绍了MGE-LDM，这是一种统一的潜在扩散框架，用于同时生成音乐、填补缺失音源和基于查询驱动的声源分离。与以前仅限于固定乐器类别的方法不同，MGE-LDM在一个单一紧凑的潜在扩散模型中学习完整混音、子混音以及单独轨道（stem）之间的联合分布。在推理阶段，MGE-LDM能够实现： 1. 完整混音生成， 2. 部分生成（即填补缺失音源），和 3. 条件于文本的任意声源提取。 通过将分离与填补任务都作为潜在空间中的条件填充任务来表述，我们的方法支持灵活、不依赖类别属性地对任何乐器来源进行操作。值得注意的是，MGE-LDM可以在不同类型的多轨数据集（如Slakh2100、MUSDB18和MoisesDB）上联合训练，而无需依赖预定义的乐器类别。音频样本可在我们的项目页面找到：此链接。
URL
https://arxiv.org/abs/2505.23305
