Abstract
Edge devices operate in constrained and varying resource settings, requiring dynamic architectures that can adapt to limitations of the available resources. To meet such demands, layer dropping ($\mathcal{LD}$) approach is typically used to transform static models into dynamic ones by skipping parts of the network along with reducing overall computational complexity. However, existing $\mathcal{LD}$ methods greatly impact the dynamic model's performance for low and high dropping cases, deteriorating the performance-computation trade-off. To this end, we propose a distillation-based layer dropping (DLD) framework that effectively combines the capabilities of knowledge distillation and $\mathcal{LD}$ in an end-to-end fashion, thereby achieving state-of-the-art performance for dynamic speech networks. Comprehensive experimentation utilizing well-known speech recognition methods, including conformer and WavLM, on three public benchmarks demonstrates the effectiveness of our framework, reducing the word error rate by $9.32\%$ and $2.25\%$ for high and no dropping cases with $33.3\%$ reduction in training time.
Abstract (translated)
边缘设备在有限和变化的资源环境中运行，需要能够适应可用资源限制的动态架构。为了满足这一需求，通常采用层掉落（$\mathcal{LD}$）方法将静态模型转换为动态模型，通过跳过网络的部分来减少整体计算复杂度。然而，现有的层掉落方法对低频和高频掉层情况下的动态模型性能影响很大，从而恶化了性能与计算量之间的权衡。为此，我们提出了一种基于蒸馏的层掉落（DLD）框架，该框架能够以端到端的方式有效地结合知识蒸馏和$\mathcal{LD}$的能力，从而在动态语音网络中实现最先进的性能。 通过使用包括Conformer和WavLM在内的知名语音识别方法，在三个公共基准上进行的全面实验展示了我们框架的有效性。对于高频掉层情况，我们的框架将词错误率降低了9.32%，而对于无掉层的情况则减少了2.25%。此外，该框架在训练时间方面也实现了33.3%的减少。
URL
https://arxiv.org/abs/2601.16117
