Abstract
Neural Audio Codecs (NACs) are widely adopted in modern speech systems, yet how they encode linguistic and paralinguistic information remains unclear. Improving the interpretability of NAC representations is critical for understanding and deploying them in sensitive applications. Hence, we employ Sparse Autoencoders (SAEs) to decompose dense NAC representations into sparse, interpretable activations. In this work, we focus on a challenging paralinguistic attribute-accent-and propose a framework to quantify NAC interpretability. We evaluate four NAC models under 16 SAE configurations using a relative performance index. Our results show that DAC and SpeechTokenizer achieve the highest interpretability. We further reveal that acoustic-oriented NACs encode accent information primarily in activation magnitudes of sparse representations, whereas phonetic-oriented NACs rely more on activation positions, and that low-bitrate EnCodec variants show higher interpretability.
Abstract (translated)
神经音频编码器（NACs）在现代语音系统中被广泛采用，但其如何编码语言及副语言信息仍不明确。提升NAC表征的可解释性，对于理解其在敏感应用中的部署至关重要。因此，我们采用稀疏自编码器（SAEs）将密集的NAC表征分解为稀疏且可解释的激活态。本研究聚焦于具有挑战性的副语言属性——口音，并提出量化NAC可解释性的框架。我们通过相对性能指数，在16种SAE配置下评估了四种NAC模型。结果表明，DAC与SpeechTokenizer具有最高的可解释性。进一步揭示：面向声学的NACs主要通过稀疏表征的激活幅度编码口音信息，而面向音素的NACs更依赖激活位置；低码率EnCodec变体表现出更高的可解释性。
URL
https://arxiv.org/abs/2603.18359
