Abstract
Knowledge distillation (KD) is a technique for transferring knowledge from complex teacher models to simpler student models, significantly enhancing model efficiency and accuracy. It has demonstrated substantial advancements in various applications including image classification, object detection, language modeling, text classification, and sentiment analysis. Recent innovations in KD methods, such as attention-based approaches, block-wise logit distillation, and decoupling distillation, have notably improved student model performance. These techniques focus on stimulus complexity, attention mechanisms, and global information capture to optimize knowledge transfer. In addition, KD has proven effective in compressing large language models while preserving accuracy, reducing computational overhead, and improving inference speed. This survey synthesizes the latest literature, highlighting key findings, contributions, and future directions in knowledge distillation to provide insights for researchers and practitioners on its evolving role in artificial intelligence and machine learning.
Abstract (translated)
知识蒸馏（KD）是一种将复杂教师模型的知识转移到更简单的学生模型的技术，显著提升了模型的效率和准确性。它在图像分类、目标检测、语言建模、文本分类以及情感分析等多种应用中取得了重要进展。最近关于KD方法的创新，例如基于注意力的方法、分块对数蒸馏（block-wise logit distillation）及解耦蒸馏（decoupling distillation），显著提高了学生模型的表现力。这些技术着重于刺激复杂度、注意机制以及全局信息捕获，以优化知识转移过程。此外，KD在压缩大型语言模型的同时保持准确性，在减少计算开销和提高推理速度方面表现出色。 本文综述了最新的研究文献，重点介绍了知识蒸馏领域的关键发现、贡献及未来发展方向，为研究人员与实践者提供了关于其在人工智能及机器学习领域内不断演变作用的见解。
URL
https://arxiv.org/abs/2504.13825
