Abstract
Text-driven speech style transfer aims to mold the intonation, pace, and timbre of a spoken utterance to match stylistic cues from text descriptions. While existing methods leverage large-scale neural architectures or pre-trained language models, the computational costs often remain high. In this paper, we present \emph{ReverBERT}, an efficient framework for text-driven speech style transfer that draws inspiration from a state space model (SSM) paradigm, loosely motivated by the image-based method of Wang and Liu~\cite{wang2024stylemamba}. Unlike image domain techniques, our method operates in the speech space and integrates a discrete Fourier transform of latent speech features to enable smooth and continuous style modulation. We also propose a novel \emph{Transformer-based SSM} layer for bridging textual style descriptors with acoustic attributes, dramatically reducing inference time while preserving high-quality speech characteristics. Extensive experiments on benchmark speech corpora demonstrate that \emph{ReverBERT} significantly outperforms baselines in terms of naturalness, expressiveness, and computational efficiency. We release our model and code publicly to foster further research in text-driven speech style transfer.
Abstract (translated)
文本驱动的语音风格转换旨在根据文本描述中的风格提示调整口语表达的情感、语速和音色。虽然现有的方法依赖大规模神经网络架构或预训练的语言模型，但计算成本依然较高。在本文中，我们提出了**ReverBERT**，这是一种高效的基于文本的语音风格转换框架，它借鉴了状态空间模型（SSM）范式的思想，受到Wang和Liu提出的基于图像的方法StyleMamba的启发。与图像领域的技术不同，我们的方法直接作用于语音领域，并通过整合潜在语音特征的离散傅里叶变换来实现平滑且连续的风格调节。我们还提出了一种新的**Transformer基SSM层**，用于连接文本风格描述和声学属性，这在保证高质量语音特性的同时显著减少了推理时间。广泛基于基准语音语料库进行的实验表明，ReverBERT在自然度、表现力及计算效率方面明显优于基础模型。我们公开发布了我们的模型和代码，以促进文本驱动的语音风格转换领域的进一步研究。
URL
https://arxiv.org/abs/2503.20992
