Abstract
MambaVoiceCloning (MVC) asks whether the conditioning path of diffusion-based TTS can be made fully SSM-only at inference, removing all attention and explicit RNN-style recurrence layers across text, rhythm, and prosody, while preserving or improving quality under controlled conditions. MVC combines a gated bidirectional Mamba text encoder, a Temporal Bi-Mamba supervised by a lightweight alignment teacher discarded after training, and an Expressive Mamba with AdaLN modulation, yielding linear-time O(T) conditioning with bounded activation memory and practical finite look-ahead streaming. Unlike prior Mamba-TTS systems that remain hybrid at inference, MVC removes attention-based duration and style modules under a fixed StyleTTS2 mel-diffusion-vocoder backbone. Trained on LJSpeech/LibriTTS and evaluated on VCTK, CSS10 (ES/DE/FR), and long-form Gutenberg passages, MVC achieves modest but statistically reliable gains over StyleTTS2, VITS, and Mamba-attention hybrids in MOS/CMOS, F0 RMSE, MCD, and WER, while reducing encoder parameters to 21M and improving throughput by 1.6x. Diffusion remains the dominant latency source, but SSM-only conditioning improves memory footprint, stability, and deployability.
Abstract (translated)
MambaVoiceCloning (MVC) 探讨能否在推理阶段将基于扩散的TTS的条件路径完全改造为仅使用SSM，从而在文本、节奏和韵律层面彻底移除所有注意力机制以及显式的RNN式循环层，同时在受控条件下保持或提升音质。MVC结合了门控双向Mamba文本编码器、由轻量级对齐教师网络（训练后丢弃）监督的时序双向Mamba，以及采用自适应层归一化调制的表现力Mamba，实现了线性时间复杂度O(T)的条件建模，并具备有界激活内存与实用的有限前瞻流式处理。与推理阶段仍为混合架构的先前Mamba-TTS系统不同，MVC在固定的StyleTTS2梅尔扩散声码器骨架上移除了基于注意力的时长与风格模块。该模型在LJSpeech/LibriTTS上训练，并在VCTK、CSS10（西班牙语/德语/法语）以及长篇古腾堡语料上进行评估。结果显示，MVC在MOS/CMOS听感评分、F0均方根误差、梅尔倒谱系数距离和词错误率上，相较于StyleTTS2、VITS及Mamba-注意力混合模型取得了适度但统计显著的提升，同时将编码器参数量压缩至2100万，并将吞吐量提高了1.6倍。扩散过程仍是主要延迟来源，但纯SSM条件路径在内存占用、稳定性与部署便利性方面具有改进优势。
URL
https://arxiv.org/abs/2604.00292
