Abstract
This paper introduces DashengTokenizer, a continuous audio tokenizer engineered for joint use in both understanding and generation tasks. Unlike conventional approaches, which train acoustic tokenizers and subsequently integrate frozen semantic knowledge, our method inverts this paradigm: we leverage frozen semantic features and inject acoustic information. In linear evaluation across 22 diverse tasks, our method outperforms previous audio codec and audio encoder baselines by a significant margin while maintaining competitive audio reconstruction quality. Notably, we demonstrate that this acoustic injection improves performance for tasks such as speech emotion recognition, music understanding, and acoustic scene classification. We further evaluate the tokenizer's generative performance on text-to-audio (TTA), text-to-music (TTM), and speech enhancement (SE). Our approach surpasses standard variational autoencoder (VAE)-based methods on TTA and TTM tasks, while its effectiveness on SE underscores its capabilities as a general-purpose audio encoder. Finally, our results challenge the prevailing assumption that VAE-based architectures are a prerequisite for audio synthesis. Checkpoints are available at this https URL.
Abstract (translated)
这篇论文介绍了DashengTokenizer，这是一种连续音频标记器，专门设计用于同时处理理解和生成任务。与传统方法不同，后者训练声学标记器并随后整合固定的语义知识，我们的方法则反其道而行之：我们利用固定化的语义特征，并注入声学信息。在包含22项多样化任务的线性评估中，我们的方法显著超越了以往的音频编解码器和音频编码器基准，在保持竞争性的音频重构质量的同时实现了这一点。值得注意的是，我们展示了这种声学注射可以提高诸如语音情感识别、音乐理解和声音场景分类等任务的表现。此外，我们在文本到音频（TTA）、文本到音乐（TTM）以及语音增强（SE）的生成性能上评估了该标记器的能力。我们的方法在TTA和TTM任务中超越了基于变分自动编码器（VAE）的标准方法，并且其在SE上的有效性进一步证实了它作为通用音频编码器的能力。最后，我们的结果挑战了认为基于VAE架构是音频合成的先决条件这一普遍假设。相关模型检查点可以在提供的链接处获取。
URL
https://arxiv.org/abs/2602.23765
