Abstract
Modern speech systems increasingly use discretized self-supervised speech representations for compression and integration with token-based models, yet their impact on emotional information remains unclear. We study how residual vector quantization (RVQ) reshapes emotional information in discrete speech representations from both representation- and task-level perspectives. Our analysis shows that aggressive compression disproportionately degrades emotion, with uneven loss across emotion classes and model architectures. To address this, we introduce emotion-aware quantization using emotion-specific and emotion-biased codebooks, improving the preservation of both hard and soft emotion perception. We further propose Emo-Q, a lightweight routed quantization method that selects emotion-specialized codebooks, improving emotion recognition performance at lower bitrates. These results highlight the importance of emotion-aware discretization for robust affective speech processing.
Abstract (translated)
现代语音系统越来越多地采用离散化自监督语音表征以实现压缩并与基于令牌的模型集成，然而这些表征对情感信息的影响尚不明确。本研究从表征层面和任务层面探讨残差向量量化（RVQ）如何重塑离散语音表征中的情感信息。分析表明，激进的压缩会不成比例地损害情感信息，且不同情感类别和模型架构的损失程度不均。为应对此问题，我们引入了基于情感感知的量化方法，使用情感专属和情感偏置码本，从而改善硬性及软性情感感知的保留效果。进一步提出Emo-Q——一种轻量级路由量化方法，通过选择情感专业化码本，在更低码率下提升情感识别性能。这些结果凸显了情感感知离散化对于鲁棒性情感语音处理的重要性。
URL
https://arxiv.org/abs/2603.21224
