Abstract
Knowledge Distillation is crucial for optimizing face recognition models for deployment in computationally limited settings, such as edge devices. Traditional KD methods, such as Raw L2 Feature Distillation or Feature Consistency loss, often fail to capture both fine-grained instance-level details and complex relational structures, leading to suboptimal performance. We propose a unified approach that integrates two novel loss functions, Instance-Level Embedding Distillation and Relation-Based Pairwise Similarity Distillation. Instance-Level Embedding Distillation focuses on aligning individual feature embeddings by leveraging a dynamic hard mining strategy, thereby enhancing learning from challenging examples. Relation-Based Pairwise Similarity Distillation captures relational information through pairwise similarity relationships, employing a memory bank mechanism and a sample mining strategy. This unified framework ensures both effective instance-level alignment and preservation of geometric relationships between samples, leading to a more comprehensive distillation process. Our unified framework outperforms state-of-the-art distillation methods across multiple benchmark face recognition datasets, as demonstrated by extensive experimental evaluations. Interestingly, when using strong teacher networks compared to the student, our unified KD enables the student to even surpass the teacher's accuracy.
Abstract (translated)
知识蒸馏对于优化面部识别模型以适应计算资源受限的环境（如边缘设备）至关重要。传统的知识蒸馏方法，例如原始L2特征蒸馏或特征一致性损失，在捕捉细粒度实例级细节和复杂关系结构方面常常表现不佳，从而导致性能次优。我们提出了一种统一的方法，该方法整合了两种新颖的损失函数：实例级嵌入蒸馏（Instance-Level Embedding Distillation）和基于关系的成对相似性蒸馏（Relation-Based Pairwise Similarity Distillation）。 实例级嵌入蒸馏通过利用动态硬挖掘策略来调整单个特征嵌入，从而增强了从挑战性样本中学习的能力。基于关系的成对相似性蒸馏则通过成对相似性的关系捕获结构化信息，并采用记忆库机制和样本挖掘策略进行优化。 这种统一框架确保了有效的实例级对齐以及样本之间几何关系的保留，进而实现了更加全面的知识蒸馏过程。广泛的实验评估表明，我们的统一框架在多个基准面部识别数据集上优于现有的知识蒸馏方法。有趣的是，当使用强大的教师网络时，学生模型通过这种方法甚至可以超越教师网络的表现准确性。
URL
https://arxiv.org/abs/2508.11376
