Abstract
Knowledge distillation (KD) has emerged as a promising technique in deep learning, typically employed to enhance a compact student network through learning from their high-performance but more complex teacher variant. When applied in the context of image super-resolution, most KD approaches are modified versions of methods developed for other computer vision tasks, which are based on training strategies with a single teacher and simple loss functions. In this paper, we propose a novel Multi-Teacher Knowledge Distillation (MTKD) framework specifically for image super-resolution. It exploits the advantages of multiple teachers by combining and enhancing the outputs of these teacher models, which then guides the learning process of the compact student network. To achieve more effective learning performance, we have also developed a new wavelet-based loss function for MTKD, which can better optimize the training process by observing differences in both the spatial and frequency domains. We fully evaluate the effectiveness of the proposed method by comparing it to five commonly used KD methods for image super-resolution based on three popular network architectures. The results show that the proposed MTKD method achieves evident improvements in super-resolution performance, up to 0.46dB (based on PSNR), over state-of-the-art KD approaches across different network structures. The source code of MTKD will be made available here for public evaluation.
Abstract (translated)
知识蒸馏（KD）在深度学习中已经成为一个有前景的技术，通常用于通过学习从其高性能但更复杂的教师变体来增强紧凑的学生网络。当应用于图像超分辨率时，大多数KD方法都是为其他计算机视觉任务开发的方法，基于单教师训练策略和简单的损失函数。在本文中，我们提出了一个名为多教师知识蒸馏（MTKD）的新框架，专门用于图像超分辨率。它利用了多个教师的优势，通过结合和增强这些教师模型的输出，从而引导紧凑学生网络的学习过程。为了获得更好的学习效果，我们还开发了一个基于小波的损失函数，该函数可以在空间和频率域中更好地优化训练过程。我们对五种最常用的KD方法进行了全面的评估，这些方法基于三种流行的网络架构。结果表明，与最先进的KD方法相比，MTKD方法在超分辨率性能上明显取得了改进，性能提高了0.46dB（基于PSNR）。MTKD的源代码将通过这里公开评估。
URL
https://arxiv.org/abs/2404.09571
