Abstract
We study timbre transfer as an inference-time editing problem for music audio. Starting from a strong pre-trained latent diffusion model, we introduce a lightweight procedure that requires no additional training: (i) a dimension-wise noise injection that targets latent channels most informative of instrument identity, and (ii) an early-step clamping mechanism that re-imposes the input's melodic and rhythmic structure during reverse diffusion. The method operates directly on audio latents and is compatible with text/audio conditioning (e.g., CLAP). We discuss design choices,analyze trade-offs between timbral change and structural preservation, and show that simple inference-time controls can meaningfully steer pre-trained models for style-transfer use cases.
Abstract (translated)
我们将音色转换视为音乐音频在推理阶段的编辑问题。基于一个强大的预训练潜在扩散模型，我们引入了一个无需额外训练的轻量级过程：(i) 针对最能体现乐器身份信息的潜在通道进行维度噪声注入；(ii) 在逆向扩散早期步骤中使用一种限制机制来重新施加输入的旋律和节奏结构。该方法直接在音频潜在空间上操作，并且与文本/音频条件（例如CLAP）兼容。我们讨论了设计选择，分析了音色变化和结构保留之间的权衡，并展示了简单的推理时间控制可以有意义地引导预训练模型以支持风格转换的应用场景。
URL
https://arxiv.org/abs/2601.01294
