Abstract
Style Transfer with Inference-Time Optimisation (ST-ITO) is a recent approach for transferring the applied effects of a reference audio to a raw audio track. It optimises the effect parameters to minimise the distance between the style embeddings of the processed audio and the reference. However, this method treats all possible configurations equally and relies solely on the embedding space, which can lead to unrealistic or biased results. We address this pitfall by introducing a Gaussian prior derived from a vocal preset dataset, DiffVox, over the parameter space. The resulting optimisation is equivalent to maximum-a-posteriori estimation. Evaluations on vocal effects transfer on the MedleyDB dataset show significant improvements across metrics compared to baselines, including a blind audio effects estimator, nearest-neighbour approaches, and uncalibrated ST-ITO. The proposed calibration reduces parameter mean squared error by up to 33% and matches the reference style better. Subjective evaluations with 16 participants confirm our method's superiority, especially in limited data regimes. This work demonstrates how incorporating prior knowledge in inference time enhances audio effects transfer, paving the way for more effective and realistic audio processing systems.
Abstract (translated)
风格转移与推理时间优化（ST-ITO）是一种最近用于将参考音频的效果转移到原始音频轨道上的方法。该方法通过调整效果参数来最小化处理后的音频的样式嵌入和参考之间的距离。然而，这种方法对所有可能的配置一视同仁，并且仅仅依赖于嵌入空间，这可能导致不现实或有偏见的结果。 为了克服这一缺陷，我们引入了一种基于DiffVox（一个声乐预设数据集）的高斯先验，在参数空间上进行设置。由此产生的优化等同于最大后验估计。在MedleyDB数据集上的声效转移评估显示，与基线方法相比，包括盲音频效果估算器、最近邻方法和未经校准的ST-ITO，我们的方法在多个指标上都有显著改进。所提出的校准减少了参数均方误差高达33%，并且更好地匹配了参考样式。 主观评价测试中，16名参与者确认了我们方法的优越性，在数据量有限的情况下尤其明显。这项工作展示了如何通过在推理时间加入先验知识来增强音频效果转移，并为更有效和现实的音频处理系统铺平道路。
URL
https://arxiv.org/abs/2505.11315
