Abstract
Text-driven voice conversion allows customization of speaker characteristics and prosodic elements using textual descriptions. However, most existing methods rely heavily on direct text-to-speech training, limiting their flexibility in controlling nuanced style elements or timbral features. In this paper, we propose a novel \textbf{Latent State-Space} approach for text-driven voice conversion (\textbf{LSS-VC}). Our method treats each utterance as an evolving dynamical system in a continuous latent space. Drawing inspiration from mamba, which introduced a state-space model for efficient text-driven \emph{image} style transfer, we adapt a loosely related methodology for \emph{voice} style transformation. Specifically, we learn a voice latent manifold where style and content can be manipulated independently by textual style prompts. We propose an adaptive cross-modal fusion mechanism to inject style information into the voice latent representation, enabling interpretable and fine-grained control over speaker identity, speaking rate, and emphasis. Extensive experiments show that our approach significantly outperforms recent baselines in both subjective and objective quality metrics, while offering smoother transitions between styles, reduced artifacts, and more precise text-based style control.
Abstract (translated)
文本驱动的语音转换技术允许通过文本描述自定义说话人的特征和韵律元素。然而，大多数现有的方法严重依赖于直接的文本到语音训练，这限制了它们在控制细微风格要素或音色特性方面的灵活性。本文提出了一种新颖的方法——**隐含状态空间**（Latent State-Space）文本驱动语音转换（LSS-VC）。我们的方法将每个语句视为连续隐含空间中的演化动力系统。借鉴mamba引入的状态空间模型，该模型用于高效的文本驱动图像风格转移，我们将其相关技术适应于声音风格变换中。 具体而言，我们学习了一个可以独立操控的音色潜在流形，其中通过文本风格提示来改变样式和内容。我们提出了一种自适应跨模态融合机制，将风格信息注入到语音潜在表示中，从而实现对说话者身份、语速和强调程度的解释性和精细控制。 大量的实验表明，我们的方法在主观和客观质量指标上均显著优于近期基线方法，并且提供了更平滑的风格转换过程，减少了伪影，并实现了更为精确的文本基础风格控制。
URL
https://arxiv.org/abs/2503.20999
