Abstract
Large language models (LLMs) are challenging to deploy for domain-specific tasks due to their massive scale. While distilling a fine-tuned LLM into a smaller student model is a promising alternative, the capacity gap between teacher and student often leads to suboptimal performance. This raises a key question: when and how can a student model match or even surpass its teacher on domain-specific tasks? In this work, we propose a novel theoretical insight: a student can outperform its teacher if its advantage on a Student-Favored Subdomain (SFS) outweighs its deficit on the Teacher-Favored Subdomain (TFS). Guided by this insight, we propose Scheduled Checkpoint Distillation (SCD), which reduces the TFS deficit by emulating the teacher's convergence process during supervised fine-tuning (SFT) on the domain task, and a sample-wise Adaptive Weighting (AW) mechanism to preserve student strengths on SFS. Experiments across diverse domain tasks--including QA, NER, and text classification in multiple languages--show that our method consistently outperforms existing distillation approaches, allowing the student model to match or even exceed the performance of its fine-tuned teacher.
Abstract (translated)
大型语言模型（LLMs）由于其巨大的规模，很难针对特定领域任务进行部署。尽管将经过微调的大型语言模型提炼成较小的学生模型是一种有前景的替代方案，但教师和学生模型之间的容量差距通常会导致次优性能。这引发了一个关键问题：在何时何地，学生模型才能匹配甚至超越其教师模型在特定领域的表现？在这项工作中，我们提出了一种新颖的理论见解：如果学生模型在其偏好的子领域（SFS）上的优势能够抵消它在教师偏好子领域（TFS）上的劣势，则该学生模型可以超过其老师的表现。根据这一洞察，我们提出了有计划检查点蒸馏（SCD），这种方法通过模拟教师模型在监督微调（SFT）过程中的收敛情况来减少TFS的差距，并且采用自适应加权机制来保持学生模型在SFS上的优势。实验结果显示，在包括问答、命名实体识别和多语言文本分类等不同领域的任务上，我们的方法始终优于现有的蒸馏方法，使得学生模型能够与甚至超越其微调后的教师模型的表现。
URL
https://arxiv.org/abs/2601.10114
