Abstract
In this paper, we address the following question: How do generic foundation models (e.g., CLIP, BLIP, LLaVa, DINO) compare against a domain-specific face recognition model (viz., AdaFace or ArcFace) on the face recognition task? Through a series of experiments involving several foundation models and benchmark datasets, we are able to report the following findings: (a) In all datasets considered, domain-specific models outperformed zero-shot foundation models. (b) The performance of zero-shot generic foundation models improves on over-segmented face images than tightly cropped faces thereby suggesting the importance of contextual clues. For example, at a False Match Rate (FMR) of 0.01%, the True Match Rate (TMR) of OpenCLIP improved from 64.97% to 81.73% on the LFW dataset as the face crop increased from 112x112 to 250x250 while the TMR of domain-specific AdaFace dropped from 99.09% to 77.31%. (c) A simple score-level fusion of a foundation model with a domain-specific FR model improved the accuracy at low FMRs. For example, the TMR of AdaFace when fused with BLIP improved from 72.64% to 83.31% at an FMR of 0.0001% on the IJB-B dataset and from 73.17% to 85.81% on the IJB-C dataset. (d) Foundation models, such as ChatGPT, can be used to impart explainability to the FR pipeline (e.g., ``Despite minor lighting and head tilt differences, the two left-profile images show high consistency in forehead slope, nose shape, chin contour...''). In some instances, foundation models are even able to resolve low-confidence decisions made by AdaFace (e.g., ``Although AdaFace assigns a low similarity score of 0.21, both images exhibit visual similarity...and the pair is likely of the same person''), thereby reiterating the importance of combining domain-specific FR models with generic foundation models in a judicious manner.
Abstract (translated)
在这篇论文中，我们探讨了以下问题：通用基础模型（如CLIP、BLIP、LLaVa、DINO）与专用于人脸识别的领域特定模型（例如AdaFace或ArcFace）在人脸识别任务上的表现如何？通过一系列涉及多个基础模型和基准数据集的实验，我们得出了如下发现： (a) 在所有考虑的数据集中，领域特定模型的表现优于零样本基础模型。 (b) 零样本通用基础模型在处理过度分割的人脸图像时比紧密裁剪的人脸图像表现出更好的性能，这表明上下文线索的重要性。例如，在错误匹配率为0.01%的情况下，OpenCLIP在LFW数据集上的真匹配率从64.97%提升到了81.73%，当面部裁剪尺寸由112x112增加到250x250时；相比之下，领域特定的AdaFace模型的真匹配率却从99.09%下降到了77.31%。 (c) 将基础模型与专有的人脸识别模型进行简单的分数级融合可以提高低错误匹配率下的准确性。例如，在IJB-B数据集上，当AdaFace与BLIP融合后，以0.0001%的FMR计算时，真匹配率从72.64%提升到了83.31%，而在IJB-C数据集上的真匹配率则从73.17%提高到85.81%。 (d) 像ChatGPT这样的基础模型可以用于为人脸识别管道赋予解释性（例如，“尽管光线和头部角度略有不同，两张侧面照在前额斜度、鼻形和下巴轮廓方面表现出高度一致性……”）。在某些情况下，基础模型甚至能够解决AdaFace做出的低置信度决策（例如，“虽然AdaFace给出的相似分数为0.21较低，但两幅图像具有视觉上的相似性……这对可能是同一个人的照片。”），这进一步强调了以明智的方式结合领域特定的人脸识别模型与通用基础模型的重要性。
URL
https://arxiv.org/abs/2507.03541
