Abstract
We present LongCat-AudioDiT, a novel, non-autoregressive diffusion-based text-to-speech (TTS) model that achieves state-of-the-art (SOTA) performance. Unlike previous methods that rely on intermediate acoustic representations such as mel-spectrograms, the core innovation of LongCat-AudioDiT lies in operating directly within the waveform latent space. This approach effectively mitigates compounding errors and drastically simplifies the TTS pipeline, requiring only a waveform variational autoencoder (Wav-VAE) and a diffusion backbone. Furthermore, we introduce two critical improvements to the inference process: first, we identify and rectify a long-standing training-inference mismatch; second, we replace traditional classifier-free guidance with adaptive projection guidance to elevate generation quality. Experimental results demonstrate that, despite the absence of complex multi-stage training pipelines or high-quality human-annotated datasets, LongCat-AudioDiT achieves SOTA zero-shot voice cloning performance on the Seed benchmark while maintaining competitive intelligibility. Specifically, our largest variant, LongCat-AudioDiT-3.5B, outperforms the previous SOTA model (Seed-TTS), improving the speaker similarity (SIM) scores from 0.809 to 0.818 on Seed-ZH, and from 0.776 to 0.797 on Seed-Hard. Finally, through comprehensive ablation studies and systematic analysis, we validate the effectiveness of our proposed modules. Notably, we investigate the interplay between the Wav-VAE and the TTS backbone, revealing the counterintuitive finding that superior reconstruction fidelity in the Wav-VAE does not necessarily lead to better overall TTS performance. Code and model weights are released to foster further research within the speech community.
Abstract (translated)
我们提出了LongCat-AudioDiT，一种新颖的、基于非自回归扩散的文本转语音（TTS）模型，实现了最先进（SOTA）的性能。与以往依赖梅尔频谱图等中间声学表示的方法不同，LongCat-AudioDiT的核心创新在于直接在波形隐空间中进行操作。该方法有效缓解了误差累积问题，并大幅简化了TTS流程，仅需一个波形变分自编码器（Wav-VAE）和一个扩散主干网络。此外，我们为推理过程引入了两项关键改进：首先，我们识别并修正了一个长期存在的训练-推理不匹配问题；其次，我们用自适应投影引导替代了传统的无分类器引导，以提升生成质量。实验结果表明，尽管没有复杂的多阶段训练流程或高质量的人工标注数据集，LongCat-AudioDiT在Seed基准上实现了SOTA的零样本声音克隆性能，同时保持了具有竞争力的可懂度。具体而言，我们最大的变体LongCat-AudioDiT-3.5B超越了之前的SOTA模型（Seed-TTS），将说话人相似度（SIM）分数从Seed-ZH上的0.809提升至0.818，从Seed-Hard上的0.776提升至0.797。最后，通过全面的消融实验和系统性分析，我们验证了所提出模块的有效性。值得注意的是，我们研究了Wav-VAE与TTS主干网络之间的相互作用，揭示了一个反直觉的发现：Wav-VAE中更高的重建保真度并不一定带来更好的整体TTS性能。我们已公开代码和模型权重，以促进语音社区内的进一步研究。
URL
https://arxiv.org/abs/2603.29339
