Abstract
Many speech enhancement (SE) methods rely on continuous representations. Recently, discrete audio tokens have been explored to enable autoregressive generation for SE. However, it remains unclear whether discretization itself consistently improves SE performance. In this paper, we introduce LL-SDR, a token-based speech enhancement framework that explicitly leverages discretization to better separate speech and noise. Our first contribution is a Variance-Ordered Residual Vector Quantizer (VO-RVQ), designed to disentangle speech and noise distributions during tokenization. Second, we propose a latent-space discriminator to better align enhanced embeddings with semantic embeddings. Experiments show that LL-SDR outperforms continuous baselines and matches the performance of autoregressive token-based approaches, while enabling lightweight, low-latency speech enhancement in both reverberant and non-reverberant noisy environments. Demos and source code are available at our project websites.
Abstract (translated)
许多语音增强（SE）方法依赖于连续表示。近期，离散音频令牌已被探索用于实现语音增强的自回归生成。然而，离散化本身是否能持续提升语音增强性能仍不明确。本文中，我们提出了LL-SDR，一个基于令牌的语音增强框架，它明确利用离散化来更好地区分语音与噪声。我们的首个贡献是方差有序残差向量量化器（VO-RVQ），旨在令牌化过程中解耦语音与噪声分布。其次，我们提出了潜在空间判别器，以更好对齐增强后的嵌入与语义嵌入。实验表明，LL-SDR优于连续基线方法，并与自回归令牌式方法性能相当，同时能在混响与非混响噪声环境中实现轻量、低延迟的语音增强。演示与源代码已发布至项目网站。
URL
https://arxiv.org/abs/2603.20242
