Abstract
Large language model (LLM)-based text-to-speech (TTS) systems achieve remarkable naturalness via autoregressive (AR) decoding, but require N sequential steps to generate N speech tokens. We present LLaDA-TTS, which replaces the AR LLM with a masked diffusion model that completes generation in a fixed number of parallel steps, decoupling inference latency from sequence length. Remarkably, using only 50 hours of fine-tuning data, we successfully transfer a pretrained AR checkpoint to the masked diffusion paradigm via bidirectional attention. At 64 steps, LLaDA-TTS achieves 0.98% CER (zh) and 1.96% WER (en) on Seed-TTS-Eval, matching the original CosyVoice 3 baseline performance while delivering a 2x LLM-stage speedup--a notable acceleration achieved despite the absence of KV cache, an optimization the AR baseline heavily relies on. Beyond acceleration, the bidirectional architecture naturally enables zero-shot speech editing--including word-level insertion, deletion, and substitution--without any additional training. Theoretically, we prove that AR-pretrained weights are near-optimal for bidirectional masked prediction under the locality property of acoustic tokens, explaining this rapid convergence. This general method modifies only the attention mask and objective, applying seamlessly to any LLM-based AR TTS system. Code and audio samples will be available at this https URL.
Abstract (translated)
基于大语言模型（LLM）的文本转语音（TTS）系统通过自回归（AR）解码实现了卓越的自然度，但需要N个连续步骤来生成N个语音标记。我们提出的LLaDA-TTS用掩码扩散模型取代了AR LLM，以固定数量的并行步骤完成生成，从而将推理延迟与序列长度解耦。值得注意的是，仅使用50小时的微调数据，我们通过双向注意力成功地将预训练的AR检查点迁移至掩码扩散范式。在64步推理下，LLaDA-TTS在Seed-TTS-Eval上实现了0.98%（中文）和1.96%（英文）的CER/WER，匹配了原始CosyVoice 3基线性能，同时实现了2倍的LLM阶段加速——这一显著加速是在未使用KV缓存（AR基线严重依赖的优化）的情况下达成的。除加速外，双向架构天然支持零样本语音编辑——包括词级插入、删除和替换——而无需额外训练。理论上，我们证明在声学标记局部性假设下，AR预训练权重对双向掩码预测近乎最优，这解释了其快速收敛。这一通用方法仅需修改注意力掩码和目标函数，即可无缝应用于任何基于LLM的AR TTS系统。代码及音频样本将在此链接发布。
URL
https://arxiv.org/abs/2603.26364
