Abstract
Transformer-based neural speech processing has achieved state-of-the-art performance. Since speech audio signals are known to be highly compressible, here we seek to accelerate neural speech transcription by time-domain signal sparsification early in the neural encoding stage, taking advantage of the interpretability of the self-attention mechanism in transformer audio encoders. With the Whisper family of models, we perform a systematic architecture search over the joint space of sparsification stage (a certain encoder layer) and compression ratio (sparsity). We found that the best resulting solutions under 1% accuracy degradation choose to sparsify the hidden state to 40-60% sparsity at an early encoding stage, and thereby achieve up to 1.6x runtime acceleration in English speech transcription tasks on Nvidia GPUs without any fine-tuning.
Abstract (translated)
基于Transformer的神经语音处理已经取得了最先进的性能。由于语音音频信号具有高度可压缩性，我们在此寻求通过在神经编码阶段早期对时域信号进行稀疏化来加速神经语音转录，并利用变压器音频编码器中自我注意机制的解释能力。使用Whisper模型系列，我们在稀疏化阶段（某个编码层）和压缩比率（稀疏度）联合空间上进行了系统的架构搜索。我们发现，在不超过1%准确率下降的情况下，最佳解决方案选择在早期编码阶段将隐藏状态稀疏化至40%-60%，从而在Nvidia GPU上实现了高达1.6倍的运行时间加速，而无需进行任何微调。
URL
https://arxiv.org/abs/2506.15912
