Abstract
The proliferation of foundation models, pretrained on large-scale unlabeled datasets, has emerged as an effective approach in creating adaptable and reusable architectures that can be leveraged for various downstream tasks using satellite observations. However, their direct application to hyperspectral remote sensing remains challenging due to inherent spectral disparities and the scarcity of available observations. In this work, we present HyperKD, a novel knowledge distillation framework that enables transferring learned representations from a teacher model into a student model for effective development of a foundation model on hyperspectral images. Unlike typical knowledge distillation frameworks, which use a complex teacher to guide a simpler student, HyperKD enables an inverse form of knowledge transfer across different types of spectral data, guided by a simpler teacher model. Building upon a Masked Autoencoder, HyperKD distills knowledge from the Prithvi foundational model into a student tailored for EnMAP hyperspectral imagery. HyperKD addresses the inverse domain adaptation problem with spectral gaps by introducing a feature-based strategy that includes spectral range-based channel alignment, spatial feature-guided masking, and an enhanced loss function tailored for hyperspectral images. HyperKD bridges the substantial spectral domain gap, enabling the effective use of pretrained foundation models for geospatial applications. Extensive experiments show that HyperKD significantly improves representation learning in MAEs, leading to enhanced reconstruction fidelity and more robust performance on downstream tasks such as land cover classification, crop type identification, and soil organic carbon prediction, underpinning the potential of knowledge distillation frameworks in remote sensing analytics with hyperspectral imagery.
Abstract (translated)
基于大规模无标签数据集训练的预训练基础模型的普及已成为利用卫星观测进行各种下游任务的有效方法。然而，这些模型直接应用于高光谱遥感仍然具有挑战性，主要原因是固有的光谱差异和可用观测数据的稀缺性。在本文中，我们提出了HyperKD，这是一种新颖的知识蒸馏框架，它能够将教师模型学习到的表示转移到学生模型中，从而有效地开发针对高光谱图像的基础模型。与典型的知识蒸馏框架不同，后者使用复杂老师指导简单的学生模型，HyperKD则使知识可以在不同类型光谱数据之间进行反向转移，并由简单的教师模型引导。基于掩码自动编码器（Masked Autoencoder），HyperKD将Prithvi基础模型的知识提炼到针对EnMAP高光谱图像优化的学生模型中。 HyperKD通过引入一种基于特征的方法来解决具有光谱差距的逆域适应问题，这种方法包括基于光谱范围的通道对齐、空间特征引导掩码和专为高光谱图像设计的增强损失函数。HyperKD弥合了显著的光谱领域差距，使预训练基础模型能够有效地用于地理空间应用中。 广泛的实验表明，HyperKD在MAEs中的表示学习方面有了显著改善，这导致了重建保真度提高以及土地覆盖分类、作物类型识别和土壤有机碳预测等下游任务上更为稳健的表现。这些结果证明了知识蒸馏框架在利用高光谱图像进行遥感分析方面的潜力。
URL
https://arxiv.org/abs/2508.09453
