Abstract
Knowledge distillation (KD) has been widely applied in semantic segmentation to compress large models, but conventional approaches primarily preserve in-domain accuracy while neglecting out-of-domain generalization, which is essential under distribution shifts. This limitation becomes more severe with the emergence of vision foundation models (VFMs): although VFMs exhibit strong robustness on unseen data, distilling them with conventional KD often compromises this ability. We propose Generalizable Knowledge Distillation (GKD), a multi-stage framework that explicitly enhances generalization. GKD decouples representation learning from task learning. In the first stage, the student acquires domain-agnostic representations through selective feature distillation, and in the second stage, these representations are frozen for task adaptation, thereby mitigating overfitting to visible domains. To further support transfer, we introduce a query-based soft distillation mechanism, where student features act as queries to teacher representations to selectively retrieve transferable spatial knowledge from VFMs. Extensive experiments on five domain generalization benchmarks demonstrate that GKD consistently outperforms existing KD methods, achieving average gains of +1.9% in foundation-to-foundation (F2F) and +10.6% in foundation-to-local (F2L) distillation. The code will be available at this https URL.
Abstract (translated)
知识蒸馏(KD)已被广泛应用于语义分割领域，用于压缩大型模型。然而，传统的做法主要保持了内部数据集的准确性，而忽略了外部泛化的关键性，在分布变化的情况下这一点尤为重要。这一局限在视觉基础模型(VFMs)出现后变得更为严重：尽管VFMs在未见数据上表现出强大的鲁棒性，但使用传统KD方法往往削弱了这种能力。我们提出了可泛化知识蒸馏(GKD)，这是一种多阶段框架，专门针对提高泛化的明确目标。GKD将表示学习与任务学习解耦。在第一阶段，学生模型通过选择性特征蒸馏获得不受领域影响的表示；在第二阶段，这些表示被冻结以进行任务适应，从而减轻对可见领域的过度拟合问题。为了进一步支持迁移，我们引入了基于查询的软蒸馏机制，在该机制中，学生特性作为查询来检索VFMs中的可迁移的空间知识。 在五个领域泛化基准上的大量实验表明，GKD始终优于现有的KD方法，在基础到基础(F2F)和基础到本地(F2L)蒸馏方面分别实现了平均1.9%和10.6%的改进。代码将在提供的链接中提供。
URL
https://arxiv.org/abs/2603.02554
