Abstract
Large-scale models (LSMs) can be an effective framework for semantic representation and understanding, thereby providing a suitable tool for designing semantic communication (SC) systems. However, their direct deployment is often hindered by high computational complexity and resource requirements. In this paper, a novel robust knowledge distillation based semantic communication (RKD-SC) framework is proposed to enable efficient and \textcolor{black}{channel-noise-robust} LSM-powered SC. The framework addresses two key challenges: determining optimal compact model architectures and effectively transferring knowledge while maintaining robustness against channel noise. First, a knowledge distillation-based lightweight differentiable architecture search (KDL-DARTS) algorithm is proposed. This algorithm integrates knowledge distillation loss and a complexity penalty into the neural architecture search process to identify high-performance, lightweight semantic encoder architectures. Second, a novel two-stage robust knowledge distillation (RKD) algorithm is developed to transfer semantic capabilities from an LSM (teacher) to a compact encoder (student) and subsequently enhance system robustness. To further improve resilience to channel impairments, a channel-aware transformer (CAT) block is introduced as the channel codec, trained under diverse channel conditions with variable-length outputs. Extensive simulations on image classification tasks demonstrate that the RKD-SC framework significantly reduces model parameters while preserving a high degree of the teacher model's performance and exhibiting superior robustness compared to existing methods.
Abstract (translated)
大规模模型（LSMs）可以作为语义表示和理解的有效框架，从而为设计语义通信（SC）系统提供合适的工具。然而，它们的直接部署通常会因高计算复杂性和资源需求而受阻。本文提出了一种基于鲁棒知识蒸馏的语义通信（RKD-SC）框架，旨在实现高效且抗信道噪声能力强的大规模模型驱动的SC。该框架解决了两个关键挑战：确定最优紧凑型模型架构和在保持对抗信道噪声能力的同时有效转移知识。 首先，提出了一种基于知识蒸馏的轻量级可微架构搜索（KDL-DARTS）算法。此算法将知识蒸馏损失与复杂性惩罚结合到神经架构搜索过程中，以识别高性能且轻量级的语义编码器架构。 其次，开发了一种新颖的两阶段鲁棒知识蒸馏（RKD）算法，用于从大规模模型（教师模型）向紧凑型编码器（学生模型）转移语义能力，并随后增强系统的稳健性。为提高对信道损害的进一步适应性，引入了具有感知信道的变压器（CAT）模块作为信道编解码器，在各种不同的信道条件下进行训练，并产生可变长度的输出。 广泛的图像分类任务模拟显示，RKD-SC框架在保持教师模型高性能的同时显著减少了模型参数数量，并且与现有方法相比表现出更优的鲁棒性。
URL
https://arxiv.org/abs/2508.02148
