Abstract
We propose Universal Speech Content Factorization (USCF), a simple and invertible linear method for extracting a low-rank speech representation in which speaker timbre is suppressed while phonetic content is preserved. USCF extends Speech Content Factorization, a closed-set voice conversion (VC) method, to an open-set setting by learning a universal speech-to-content mapping via least-squares optimization and deriving speaker-specific transformations from only a few seconds of target speech. We show through embedding analysis that USCF effectively removes speaker-dependent variation. As a zero-shot VC system, USCF achieves competitive intelligibility, naturalness, and speaker similarity compared to methods that require substantially more target-speaker data or additional neural training. Finally, we demonstrate that as a training-efficient timbre-disentangled speech feature, USCF features can serve as the acoustic representation for training timbre-prompted text-to-speech models. Speech samples and code are publicly available.
Abstract (translated)
我们提出了通用语音内容因子化（USCF），这是一种简单且可逆的线性方法，用于提取一种低秩语音表示，在这种表示中，说话人的音色被抑制而语音内容得以保留。USCF通过最小二乘优化学习一个通用的语音到内容映射，并仅使用几秒钟的目标说话人音频来推导出特定说话者的转换，从而将封闭集内的声音转换方法——语音内容因子化扩展到了开放集中。我们通过嵌入分析展示了USCF能够有效地移除与说话人相关的变异性。作为零样本声音转换系统，USCF在可懂度、自然度和说话人相似性方面取得了优于需要大量目标说话人数据或额外神经网络训练的方法的竞争力结果。最后，我们演示了USCF作为一种高效的音色解耦语音特征，在训练由音色调用的文本到语音模型中可以作为声学表示使用。语音样本和代码已公开提供。
URL
https://arxiv.org/abs/2603.08977
