Abstract
Large language models (LLMs) have significantly advanced audio processing through audio codecs that convert audio into discrete tokens, enabling the application of language modelling techniques to audio data. However, traditional codecs often operate at high bitrates or within narrow domains such as speech and lack the semantic clues required for efficient language modelling. Addressing these challenges, we introduce SemantiCodec, a novel codec designed to compress audio into fewer than a hundred tokens per second across diverse audio types, including speech, general audio, and music, without compromising quality. SemantiCodec features a dual-encoder architecture: a semantic encoder using a self-supervised AudioMAE, discretized using k-means clustering on extensive audio data, and an acoustic encoder to capture the remaining details. The semantic and acoustic encoder outputs are used to reconstruct audio via a diffusion-model-based decoder. SemantiCodec is presented in three variants with token rates of 25, 50, and 100 per second, supporting a range of ultra-low bit rates between 0.31 kbps and 1.43 kbps. Experimental results demonstrate that SemantiCodec significantly outperforms the state-of-the-art Descript codec on reconstruction quality. Our results also suggest that SemantiCodec contains significantly richer semantic information than all evaluated audio codecs, even at significantly lower bitrates. Our code and demos are available at this https URL.
Abstract (translated)
大语言模型（LLMs）通过将音频转换为离散 tokens 的音频编码器显著提高了音频处理能力，使得将语言建模技术应用于音频数据成为可能。然而，传统的编码器通常操作在高速位率或狭窄的领域内，如语音，缺乏进行高效语言建模所需的语义线索。为解决这些挑战，我们引入了 SemantiCodec，一种专为将音频压缩成不到100个 tokens per second 的音频编码器，包括语音、通用音频和音乐，同时不牺牲质量。SemantiCodec 具有双重编码器架构：一个使用自监督的 AudioMAE 的语义编码器，通过扩展音频数据上的 k-means 聚类进行离散化，以及一个声学编码器来捕捉剩余细节。语义和声学编码器的输出被用于通过扩散模型解码器重构音频。SemantiCodec 推出了三种版本，具有不同的 token rate，支持在 0.31 kbps 和 1.43 kbps 之间运行的极低比特率。实验结果表明，SemantiCodec 在重构质量方面显著优于最先进的 Descript 编码器。我们的结果还表明，即使在显著较低的比特率下，SemantiCodec 也包含比所有评估音频编码器更丰富的语义信息。我们的代码和演示文稿可以从此链接获取。
URL
https://arxiv.org/abs/2405.00233
