Abstract
Foundation models such as CLIP have demonstrated exceptional zero- and few-shot transfer capabilities across diverse vision tasks. However, when fine-tuned for highly specialized biometric tasks, face recognition (FR), morphing attack detection (MAD), and presentation attack detection (PAD), these models may suffer from over-specialization. Thus, they may lose one of their foundational strengths, cross-domain generalization. In this work, we systematically quantify these trade-offs by evaluating three instances of CLIP fine-tuned for FR, MAD, and PAD. We evaluate each adapted model as well as the original CLIP baseline on 14 general vision datasets under zero-shot and linear-probe protocols, alongside common FR, MAD, and PAD benchmarks. Our results indicate that fine-tuned models suffer from over-specialization, especially when fine-tuned for complex tasks of FR. Also, our results pointed out that task complexity and classification head design, multi-class (FR) vs. binary (MAD and PAD), correlate with the degree of catastrophic forgetting. The FRoundation model with the ViT-L backbone outperforms other approaches on the large-scale FR benchmark IJB-C, achieving an improvement of up to 58.52%. However, it experiences a substantial performance drop on ImageNetV2, reaching only 51.63% compared to 69.84% achieved by the baseline CLIP model. Moreover, the larger CLIP architecture consistently preserves more of the model's original generalization ability than the smaller variant, indicating that increased model capacity may help mitigate over-specialization.
Abstract (translated)
基础模型（如CLIP）在各种视觉任务中展示了卓越的零样本和少量样本迁移能力。然而，当这些模型被微调用于高度专业化的生物识别任务（例如面部识别、变形攻击检测和呈现攻击检测）时，它们可能会出现过度专业化的问题，从而丧失其跨领域泛化的基本优势。 在这项工作中，我们系统地量化了这种权衡，通过评估三组CLIP模型在经过面部识别、形态攻击检测和呈现攻击检测微调后的性能。我们在14个通用视觉数据集以及常见的面部识别、变形攻击检测和呈现攻击检测基准上，使用零样本和线性探测协议对每个适应的模型及原始CLIP基线进行了评估。 我们的结果显示，经过微调的模型会遭受过度专业化的问题，尤其是当它们用于复杂的面部识别任务时。此外，我们的研究还表明，任务复杂性和分类头的设计（多类 vs 二元）与灾难性遗忘的程度相关联。带有ViT-L骨干网络的基础模型在大规模面部识别基准IJB-C上表现优于其他方法，改进幅度高达58.52%。然而，在ImageNetV2数据集上，它的性能显著下降，仅达到51.63%，相比之下，原始的CLIP基线模型则达到了69.84%。 此外，较大的CLIP架构相比较小的变体在一定程度上更好地保持了模型原有的泛化能力，这表明增加模型容量可能有助于减轻过度专业化的问题。
URL
https://arxiv.org/abs/2509.14921
