Abstract
Multimodal Large Language Models (MLLMs) have recently been explored as face verification systems that determine whether two face images are of the same person. Unlike dedicated face recognition systems, MLLMs approach this task through visual prompting and rely on general visual and reasoning abilities. However, the demographic fairness of these models remains largely unexplored. In this paper, we present a benchmarking study that evaluates nine open-source MLLMs from six model families, ranging from 2B to 8B parameters, on the IJB-C and RFW face verification protocols across four ethnicity groups and two gender groups. We measure verification accuracy with the Equal Error Rate and True Match Rate at multiple operating points per demographic group, and we quantify demographic disparity with four FMR-based fairness metrics. Our results show that FaceLLM-8B, the only face-specialised model in our study, substantially outperforms general-purpose MLLMs on both benchmarks. The bias patterns we observe differ from those commonly reported for traditional face recognition, with different groups being most affected depending on the benchmark and the model. We also note that the most accurate models are not necessarily the fairest and that models with poor overall accuracy can appear fair simply because they produce uniformly high error rates across all demographic groups.
Abstract (translated)
多模态大语言模型（MLLMs）近期被探索用于面部验证系统，以判断两张人脸图像是否属于同一人。与专用面部识别系统不同，MLLMs通过视觉提示处理此任务，并依赖通用的视觉与推理能力。然而，这些模型的人口统计公平性尚未得到充分研究。本文提出一项基准研究，在IJB-C和RFW面部验证协议上，评估了六个模型家族中的九个开源MLLMs（参数规模从2B至8B），覆盖四个种族群体和两个性别群体。我们采用等误率（EER）和多个操作点下的真匹配率（TMR）按人口统计组别测量验证准确率，并使用四个基于误识率（FMR）的公平性指标量化人口统计差异。研究结果表明：唯一的面部专用模型FaceLLM-8B在两个基准测试中均显著优于通用MLLMs。观察到的偏差模式与传统面部识别常见报告不同——不同群体受影响程度因基准测试和模型而异。同时发现，最准确的模型未必最公平，而整体准确率较低的模型可能因在所有群体中产生均匀的高错误率而显得"公平"。
URL
https://arxiv.org/abs/2603.25613
