Abstract
Semantic information refers to the meaning conveyed through words, phrases, and contextual relationships within a given linguistic structure. Humans can leverage semantic information, such as familiar linguistic patterns and contextual cues, to reconstruct incomplete or masked speech signals in noisy environments. However, existing speech enhancement (SE) approaches often overlook the rich semantic information embedded in speech, which is crucial for improving intelligibility, speaker consistency, and overall quality of enhanced speech signals. To enrich the SE model with semantic information, we employ language models as an efficient semantic learner and propose a comprehensive framework tailored for language model-based speech enhancement, called \textit{GenSE}. Specifically, we approach SE as a conditional language modeling task rather than a continuous signal regression problem defined in existing works. This is achieved by tokenizing speech signals into semantic tokens using a pre-trained self-supervised model and into acoustic tokens using a custom-designed single-quantizer neural codec model. To improve the stability of language model predictions, we propose a hierarchical modeling method that decouples the generation of clean semantic tokens and clean acoustic tokens into two distinct stages. Moreover, we introduce a token chain prompting mechanism during the acoustic token generation stage to ensure timbre consistency throughout the speech enhancement process. Experimental results on benchmark datasets demonstrate that our proposed approach outperforms state-of-the-art SE systems in terms of speech quality and generalization capability.
Abstract (translated)
语义信息指的是通过词汇、短语以及语言结构中的上下文关系传达的意义。人类可以利用这些语义信息，例如熟悉的语言模式和上下文线索，在嘈杂环境中重建不完整或被屏蔽的语音信号。然而，现有的语音增强（SE）方法往往忽视了嵌入在语音中的丰富语义信息，这对于提高增强语音信号的理解度、说话者一致性及整体质量至关重要。为了使SE模型更加丰富地利用语义信息，我们采用语言模型作为有效的语义学习工具，并提出了一种基于语言模型的语音增强框架，称为\textit{GenSE}。 具体而言，我们将SE视为一个条件语言建模任务，而不是现有研究定义中的连续信号回归问题。这通过使用预训练的自监督模型将语音信号转换为语义标记以及采用定制设计的单量化器神经编解码模型将语音信号转换为声学标记来实现。为了提高语言模型预测的稳定性，我们提出了一种分层建模方法，该方法将生成干净语义标记和干净声学标记的过程分为两个独立阶段。此外，在声学标记生成阶段中，我们引入了一个令牌链提示机制以确保在整个语音增强过程中音色的一致性。 在基准数据集上的实验结果表明，我们的方法在语音质量和泛化能力方面均优于现有的最先进的SE系统。
URL
https://arxiv.org/abs/2502.02942
