Face_Recognition
-
Adv-CPG: A Customized Portrait Generation Framework with Facial Adversarial Attacks
2025-03-11 10:34:57Junying Wang, Hongyuan Zhang, Yuan YuanAbstract
Recent Customized Portrait Generation (CPG) methods, taking a facial image and a textual prompt as inputs, have attracted substantial attention. Although these methods generate high-fidelity portraits, they fail to prevent the generated portraits from being tracked and misused by malicious face recognition systems. To address this, this paper proposes a Customized Portrait Generation framework with facial Adversarial attacks (Adv-CPG). Specifically, to achieve facial privacy protection, we devise a lightweight local ID encryptor and an encryption enhancer. They implement progressive double-layer encryption protection by directly injecting the target identity and adding additional identity guidance, respectively. Furthermore, to accomplish fine-grained and personalized portrait generation, we develop a multi-modal image customizer capable of generating controlled fine-grained facial features. To the best of our knowledge, Adv-CPG is the first study that introduces facial adversarial attacks into CPG. Extensive experiments demonstrate the superiority of Adv-CPG, e.g., the average attack success rate of the proposed Adv-CPG is 28.1% and 2.86% higher compared to the SOTA noise-based attack methods and unconstrained attack methods, respectively.
Abstract (translated)
最近定制化肖像生成（CPG）方法，通过面部图像和文本提示作为输入，引起了广泛关注。尽管这些方法能够生成高保真的面部画像，但它们无法防止生成的画像被恶意人脸识别系统追踪并滥用。为了解决这个问题，本文提出了一种结合了面部对抗攻击的定制化肖像生成框架（Adv-CPG）。为了实现面部隐私保护，我们设计了一个轻量级的身份加密器和一个增强器。身份加密器通过直接注入目标身份信息来实施渐进式的双层加密保护；而增强器则是通过添加额外的身份指导来加强这一过程。 此外，为了完成精细且个性化的肖像生成，我们开发了一种多模态图像定制器，能够生成受控的细粒度面部特征。据我们所知，Adv-CPG是第一个将面部对抗攻击引入到CPG中的研究工作。广泛的实验展示了Adv-CPG的优势，例如：提出的Adv-CPG方法平均攻击成功率达到了28.1%，比最先进的基于噪声的方法高出2.86%。 翻译成中文如下： 最近定制化肖像生成（Customized Portrait Generation, CPG）方法因其能够利用面部图像和文本提示作为输入而吸引了大量关注。尽管这些方法可以生成高保真的面部画像，但它们无法防止生成的画像被恶意的人脸识别系统追踪并滥用。为了解决这个问题，本文提出了一种结合了面部对抗攻击（Adversarial attacks, Adv）的定制化肖像生成框架（Adv-CPG）。为了实现面部隐私保护，我们设计了一个轻量级的身份加密器和一个增强器。身份加密器通过直接注入目标身份信息来实施渐进式的双层加密保护；而增强器则是通过添加额外的身份指导来加强这一过程。 此外，为了完成精细且个性化的肖像生成，我们开发了一种多模态图像定制器，能够生成受控的细粒度面部特征。据我们所知，Adv-CPG是第一个将面部对抗攻击引入到CPG中的研究工作。广泛的实验展示了Adv-CPG的优势，例如：提出的Adv-CPG方法平均攻击成功率达到了28.1%，比最先进的基于噪声的方法高出2.86%。
URL
https://arxiv.org/abs/2503.08269
PDF
-
FairDeFace: Evaluating the Fairness and Adversarial Robustness of Face Obfuscation Methods
2025-03-11 01:49:43Seyyed Mohammad Sadegh Moosavi Khorzooghi, Poojitha Thota, Mohit Singhal, Abolfazl Asudeh, Gautam Das, Shirin NilizadehAbstract
The lack of a common platform and benchmark datasets for evaluating face obfuscation methods has been a challenge, with every method being tested using arbitrary experiments, datasets, and metrics. While prior work has demonstrated that face recognition systems exhibit bias against some demographic groups, there exists a substantial gap in our understanding regarding the fairness of face obfuscation methods. Providing fair face obfuscation methods can ensure equitable protection across diverse demographic groups, especially since they can be used to preserve the privacy of vulnerable populations. To address these gaps, this paper introduces a comprehensive framework, named FairDeFace, designed to assess the adversarial robustness and fairness of face obfuscation methods. The framework introduces a set of modules encompassing data benchmarks, face detection and recognition algorithms, adversarial models, utility detection models, and fairness metrics. FairDeFace serves as a versatile platform where any face obfuscation method can be integrated, allowing for rigorous testing and comparison with other state-of-the-art methods. In its current implementation, FairDeFace incorporates 6 attacks, and several privacy, utility and fairness metrics. Using FairDeFace, and by conducting more than 500 experiments, we evaluated and compared the adversarial robustness of seven face obfuscation methods. This extensive analysis led to many interesting findings both in terms of the degree of robustness of existing methods and their biases against some gender or racial groups. FairDeFace also uses visualization of focused areas for both obfuscation and verification attacks to show not only which areas are mostly changed in the obfuscation process for some demographics, but also why they failed through focus area comparison of obfuscation and verification.
Abstract (translated)
缺乏一个用于评估面部模糊方法的通用平台和基准数据集一直是一个挑战，每个方法都使用任意的实验、数据集和指标进行测试。尽管先前的工作表明面部识别系统对某些人口群体存在偏见，但我们对于面部模糊方法公平性的理解仍存在很大差距。提供公平的面部模糊方法可以确保跨多种人口群体的平等保护，尤其是因为它们可用于保护脆弱人群的隐私。为解决这些差距，本文介绍了一个全面的框架，名为FairDeFace，旨在评估面部模糊方法的对抗鲁棒性和公平性。该框架引入了一系列模块，涵盖数据基准、面部检测和识别算法、对抗模型、效用检测模型以及公平性指标。FairDeFace作为任何面部模糊方法都可以整合的多功能平台，允许对其进行严格测试并与其他最先进的方法进行比较。在其目前实施中，FairDeFace集成了6种攻击方式及多种隐私、效用和公平度量标准。通过使用FairDeFace，并进行了超过500次实验，我们评估并对比了七种面部模糊方法的对抗鲁棒性。这一广泛的分析在现有方法的鲁棒程度以及它们对某些性别或种族群体的偏见方面得出了许多有趣的发现。此外，FairDeFace还利用针对模糊和验证攻击的重点区域可视化技术来展示，在某种程度上不仅展示了某些人口群体在模糊过程中变化最大的区域，而且还通过对比重点区域揭示了为什么这些方法会失败。
URL
https://arxiv.org/abs/2503.08731
PDF
-
AttFC: Attention Fully-Connected Layer for Large-Scale Face Recognition with One GPU
2025-03-10 01:59:11Zhuowen Zheng, Yain-Whar Si, Xiaochen Yuan, Junwei Duan, Ke Wang, Xiaofan Li, Xinyuan Zhang, Xueyuan GongAbstract
Nowadays, with the advancement of deep neural networks (DNNs) and the availability of large-scale datasets, the face recognition (FR) model has achieved exceptional performance. However, since the parameter magnitude of the fully connected (FC) layer directly depends on the number of identities in the dataset. If training the FR model on large-scale datasets, the size of the model parameter will be excessively huge, leading to substantial demand for computational resources, such as time and memory. This paper proposes the attention fully connected (AttFC) layer, which could significantly reduce computational resources. AttFC employs an attention loader to generate the generative class center (GCC), and dynamically store the class center with Dynamic Class Container (DCC). DCC only stores a small subset of all class centers in FC, thus its parameter count is substantially less than the FC layer. Also, training face recognition models on large-scale datasets with one GPU often encounter out-of-memory (OOM) issues. AttFC overcomes this and achieves comparable performance to state-of-the-art methods.
Abstract (translated)
如今，随着深度神经网络（DNN）的发展和大规模数据集的可用性，人脸识别（FR）模型已经取得了卓越的成绩。然而，由于全连接层（FC）参数的数量直接依赖于数据集中身份的数量，当在大规模数据集上训练FR模型时，模型参数的数量会变得极其庞大，从而对计算资源的需求大幅增加，如时间和内存等。本文提出了注意全连接（AttFC）层，该方法可以显著减少计算资源需求。AttFC使用注意力加载器来生成生成类中心（GCC），并利用动态类别容器（DCC）动态存储类中心。DCC仅存储全连接层中所有类中心的一个小子集，因此其参数数量远小于传统的全连接层。此外，在具有大规模数据集的情况下，用单个GPU训练人脸识别模型常常会遇到内存不足的问题。AttFC解决了这一问题，并且在性能上与最先进的方法相当。
URL
https://arxiv.org/abs/2503.06839
PDF
-
MSConv: Multiplicative and Subtractive Convolution for Face Recognition
2025-03-08 12:18:29Si Zhou, Yain-Whar Si, Xiaochen Yuan, Xiaofan Li, Xiaoxiang Liu, Xinyuan Zhang, Cong Lin, Xueyuan GongAbstract
In Neural Networks, there are various methods of feature fusion. Different strategies can significantly affect the effectiveness of feature representation, consequently influencing the ability of model to extract representative and discriminative features. In the field of face recognition, traditional feature fusion methods include feature concatenation and feature addition. Recently, various attention mechanism-based fusion strategies have emerged. However, we found that these methods primarily focus on the important features in the image, referred to as salient features in this paper, while neglecting another equally important set of features for image recognition tasks, which we term differential features. This may cause the model to overlook critical local differences when dealing with complex facial samples. Therefore, in this paper, we propose an efficient convolution module called MSConv (Multiplicative and Subtractive Convolution), designed to balance the learning of model about salient and differential features. Specifically, we employ multi-scale mixed convolution to capture both local and broader contextual information from face images, and then utilize Multiplication Operation (MO) and Subtraction Operation (SO) to extract salient and differential features, respectively. Experimental results demonstrate that by integrating both salient and differential features, MSConv outperforms models that only focus on salient features.
Abstract (translated)
在神经网络中，存在多种特征融合的方法。不同的策略会显著影响特征表示的有效性，从而影响模型提取代表性及区分性特征的能力。在人脸识别领域，传统的特征融合方法包括特征级联和特征相加。近期，基于注意力机制的融合策略也开始涌现。然而，我们发现这些方法主要关注图像中的重要特征（本文称为显著特征），而忽视了另一组同样重要的、用于图像识别任务的特征，我们将这组特征命名为差异性特征。这可能导致模型在处理复杂面部样本时忽略关键局部差异。因此，在本文中，我们提出了一种高效的卷积模块——MSConv（乘法和减法卷积），旨在平衡模型对显著性和差异性特征的学习。具体而言，我们采用多尺度混合卷积来捕捉人脸图像中的局部信息及更广泛的上下文，并通过乘法运算(MO)和减法运算(SO)分别提取显著性和差异性特征。实验结果表明，通过融合这两类特征，MSConv的表现优于仅关注显著性特征的模型。
URL
https://arxiv.org/abs/2503.06187
PDF
-
AdaSin: Enhancing Hard Sample Metrics with Dual Adaptive Penalty for Face Recognition
2025-03-05 14:11:13Qiqi Guo, Zhuowen Zheng, Guanghua Yang, Zhiquan Liu, Xiaofan Li, Jianqing Li, Jinyu Tian, Xueyuan GongAbstract
In recent years, the emergence of deep convolutional neural networks has positioned face recognition as a prominent research focus in computer vision. Traditional loss functions, such as margin-based, hard-sample mining-based, and hybrid approaches, have achieved notable performance improvements, with some leveraging curriculum learning to optimize training. However, these methods often fall short in effectively quantifying the difficulty of hard samples. To address this, we propose Adaptive Sine (AdaSin) loss function, which introduces the sine of the angle between a sample's embedding feature and its ground-truth class center as a novel difficulty metric. This metric enables precise and effective penalization of hard samples. By incorporating curriculum learning, the model dynamically adjusts classification boundaries across different training stages. Unlike previous adaptive-margin loss functions, AdaSin introduce a dual adaptive penalty, applied to both the positive and negative cosine similarities of hard samples. This design imposes stronger constraints, enhancing intra-class compactness and inter-class separability. The combination of the dual adaptive penalty and curriculum learning is guided by a well-designed difficulty metric. It enables the model to focus more effectively on hard samples in later training stages, and lead to the extraction of highly discriminative face features. Extensive experiments across eight benchmarks demonstrate that AdaSin achieves superior accuracy compared to other state-of-the-art methods.
Abstract (translated)
近年来，深度卷积神经网络的出现将人脸识别确立为计算机视觉领域的一个重要研究焦点。传统的损失函数，如基于边距、基于难例挖掘和混合方法等，已经实现了显著的性能改进，其中一些通过课程学习来优化训练过程。然而，这些方法往往无法有效地量化难样本的难度。为此，我们提出了自适应正弦（AdaSin）损失函数，该函数引入了样本嵌入特征与其真实类别中心之间的夹角正弦值作为新的难度度量标准。这一指标能够精准且有效地区分并惩罚难样本。通过结合课程学习，模型能够在不同的训练阶段动态调整分类边界。 与之前的自适应边距损失函数不同，AdaSin引入了一种双重自适应惩罚机制，分别应用于硬样本的正余弦相似性和负余弦相似性上。这种设计施加了更强的约束条件，增强了类内紧致度和类间分离度。通过精心设计的难度度量标准来指导双重自适应惩罚与课程学习相结合的方式，使得模型能够更有效地关注后期训练阶段中的难样本，并最终提取出高度判别性的面部特征。 在八个基准数据集上的广泛实验表明，AdaSin相比其他最先进的方法取得了更为优越的准确率。
URL
https://arxiv.org/abs/2503.03528
PDF
-
Simulation of prosthetic vision with PRIMA system and enhancement of face representation
2025-03-01 17:54:31Jungyeon Park, Anna Kochnev Goldstein, Yueming Zhou, Nathan Jensen, Daniel PalankerAbstract
Objective. Patients implanted with the PRIMA photovoltaic subretinal prosthesis in geographic atrophy report form vision with the average acuity matching the 100um pixel size. Although this remarkable outcome enables them to read and write, they report difficulty with perceiving faces. This paper provides a novel, non-pixelated algorithm for simulating prosthetic vision the way it is experienced by PRIMA patients, compares the algorithm's predictions to clinical perceptual outcomes, and offers computer vision and machine learning (ML) methods to improve face representation. Approach. Our simulation algorithm integrates a grayscale filter, spatial resolution filter, and contrast filter. This accounts for the limited sampling density of the retinal implant, as well as the reduced contrast sensitivity of prosthetic vision. Patterns of Landolt C and faces created using this simulation algorithm are compared to reports from actual PRIMA users. To recover the facial features lost in prosthetic vision, we apply an ML facial landmarking model as well as contrast adjusting tone curves to the face image prior to its projection onto the implant. Main results. Simulated prosthetic vision matches the maximum letter acuity observed in clinical studies as well as patients' subjective descriptions. Application of the inversed contrast filter helps preserve the contrast in prosthetic vision. Identification of the facial features using an ML facial landmarking model and accentuating them further improve face representation. Significance. Spatial and contrast constraints of prosthetic vision limit resolvable features and degrade natural images. ML based methods and contrast adjustments mitigate some limitations and improve face representation. Even though higher spatial resolution can be expected with implants having smaller pixels, contrast enhancement still remains essential for face recognition.
Abstract (translated)
目标。植入PRIMA光伏亚视网膜假体的患者报告称，他们的视力与100微米像素大小匹配的平均清晰度相似。尽管这种显著的结果使他们能够阅读和书写，但他们表示在感知面部方面存在困难。本文提供了一种用于模拟假肢视觉的新算法（该算法类似于PRIMA患者体验到的视觉），将此算法预测结果与临床感知效果进行比较，并提出计算机视觉和机器学习(ML)方法以改善面部表现。 方法。我们的模拟算法结合了灰度滤波器、空间分辨率滤波器和对比度滤波器，这可以解释视网膜植入物有限的采样密度以及假肢视力减少的对比敏感度。使用此仿真算法创建的Landolt C图案及面孔与实际PRIMA用户报告进行了比较。为了恢复在假肢视觉中丢失的脸部特征，我们在将图像投影到假体之前应用了机器学习面部标记模型和调整对比度的色调曲线。 主要结果。模拟的假体视觉匹配临床研究观察到的最大字母清晰度以及患者的主观描述。使用反向对比滤波器有助于保留假体视觉中的对比度。使用ML面部地标模型识别面部特征并进一步强调它们可以改善面部表现。 意义。由于空间和对比限制，假体视觉在可分辨的特性方面受到限制，并且会降解自然图像。基于机器学习的方法和对比调整能够缓解一些局限性并改善面部表示。尽管具有更小像素的植入物可能具有更高的空间分辨率，但对比度增强仍然是面部识别中的关键因素。
URL
https://arxiv.org/abs/2503.11677
PDF
-
TSDW: A Tri-Stream Dynamic Weight Network for Cloth-Changing Person Re-Identification
2025-03-01 13:04:49Ruiqi He, Zihan Wang, Xiang ZhouAbstract
Cloth-Changing Person Re-identification (CC-ReID) aims to solve the challenge of identifying individuals across different temporal-spatial scenarios, viewpoints, and clothing variations. This field is gaining increasing attention in big data research and public security domains. Existing ReID research primarily relies on face recognition, gait semantic recognition, and clothing-irrelevant feature identification, which perform relatively well in scenarios with high-quality clothing change videos and images. However, these approaches depend on either single features or simple combinations of multiple features, making further performance improvements difficult. Additionally, limitations such as missing facial information, challenges in gait extraction, and inconsistent camera parameters restrict the broader application of CC-ReID. To address the above limitations, we innovatively propose a Tri-Stream Dynamic Weight Network (TSDW) that requires only images. This dynamic weighting network consists of three parallel feature streams: facial features, head-limb features, and global features. Each stream specializes in extracting its designated features, after which a gating network dynamically fuses confidence levels. The three parallel feature streams enhance recognition performance and reduce the impact of any single feature failure, thereby improving model robustness. Extensive experiments on benchmark datasets (e.g., PRCC, Celeb-reID, VC-Clothes) demonstrate that our method significantly outperforms existing state-of-the-art approaches.
Abstract (translated)
服装变化人员重识别（CC-ReID）旨在解决跨不同时空场景、视角和着装变化时个体身份的识别挑战。这一领域在大数据研究和公共安全领域正日益受到关注。现有的ReID研究主要依赖于面部识别、步态语义识别以及与衣物无关特征的识别，这些方法在高质量的服装变化视频和图像场景中表现相对较好。然而，这些方法要么仅依靠单一特征，要么简单地组合多个特征，这使得进一步提升性能变得困难。此外，缺乏面部信息、步态提取挑战以及摄像机参数不一致等问题也限制了CC-ReID更广泛的应用。 为了解决上述限制，我们创新性地提出了一种三流动态权重网络（TSDW），该方法仅需要图像作为输入。这一动态加权网络由三个并行的特征流组成：面部特征、头部与肢体特征以及全局特征。每个特征流专门负责提取其指定的信息，在此之后通过一个门控网络动态融合置信度级别。这三个平行特征流增强了识别性能，并减少了任何单一特征失效对整体性能的影响，从而提高了模型的鲁棒性。 在基准数据集（如PRCC、Celeb-reID、VC-Clothes）上的广泛实验表明，我们提出的方法显著优于现有最先进的方法。
URL
https://arxiv.org/abs/2503.00477
PDF
-
UIFace: Unleashing Inherent Model Capabilities to Enhance Intra-Class Diversity in Synthetic Face Recognition
2025-02-27 06:22:18Xiao Lin, Yuge Huang, Jianqing Xu, Yuxi Mi, Shuigeng Zhou, Shouhong DingAbstract
Face recognition (FR) stands as one of the most crucial applications in computer vision. The accuracy of FR models has significantly improved in recent years due to the availability of large-scale human face datasets. However, directly using these datasets can inevitably lead to privacy and legal problems. Generating synthetic data to train FR models is a feasible solution to circumvent these issues. While existing synthetic-based face recognition methods have made significant progress in generating identity-preserving images, they are severely plagued by context overfitting, resulting in a lack of intra-class diversity of generated images and poor face recognition performance. In this paper, we propose a framework to Unleash Inherent capability of the model to enhance intra-class diversity for synthetic face recognition, shortened as UIFace. Our framework first trains a diffusion model that can perform sampling conditioned on either identity contexts or a learnable empty context. The former generates identity-preserving images but lacks variations, while the latter exploits the model's intrinsic ability to synthesize intra-class-diversified images but with random identities. Then we adopt a novel two-stage sampling strategy during inference to fully leverage the strengths of both types of contexts, resulting in images that are diverse as well as identitypreserving. Moreover, an attention injection module is introduced to further augment the intra-class variations by utilizing attention maps from the empty context to guide the sampling process in ID-conditioned generation. Experiments show that our method significantly surpasses previous approaches with even less training data and half the size of synthetic dataset. The proposed UIFace even achieves comparable performance with FR models trained on real datasets when we further increase the number of synthetic identities.
Abstract (translated)
人脸识别（FR）是计算机视觉领域中最重要的应用之一。近年来，由于大规模人脸数据集的可用性，人脸识别模型的准确性得到了显著提高。然而，直接使用这些数据集不可避免地会导致隐私和法律问题。生成合成数据以训练人脸识别模型是一种可行的方法来避免这些问题。虽然现有的基于合成的人脸识别方法在生成身份保持图像方面已经取得了重大进展，但它们仍然严重受到上下文过拟合的影响，导致生成的图像缺乏类内多样性，并且面部识别性能较差。 为此，在本文中我们提出了一种框架，旨在通过增强合成人脸数据的类内多样性来提升模型的内在能力，简称UIFace。我们的框架首先训练一个扩散模型，该模型可以在身份上下文或可学习的空上下文中进行采样。前者可以生成身份保持图像但缺乏变化性，后者则能够利用模型固有的能力以随机的身份生成类内多样化的图像。然后我们在推理阶段采用一种新颖的两阶段采样策略，充分利用这两种上下文的优点，从而产生多样化且身份保持的图像。 此外，我们引入了一个注意力注入模块，通过使用空上下文的注意力图来指导身份条件下的生成过程，进一步增强类内变化性。实验结果显示，即使在训练数据更少和合成数据集规模减半的情况下，我们的方法仍然显著优于先前的方法。当增加更多的合成身份数量时，UIFace甚至能够实现与基于真实数据集的人脸识别模型相当的性能。
URL
https://arxiv.org/abs/2502.19803
PDF
-
SAP-DIFF: Semantic Adversarial Patch Generation for Black-Box Face Recognition Models via Diffusion Models
2025-02-27 02:57:29Mingsi Wang, Shuaiyin Yao, Chang Yue, Lijie Zhang, Guozhu MengAbstract
Given the need to evaluate the robustness of face recognition (FR) models, many efforts have focused on adversarial patch attacks that mislead FR models by introducing localized perturbations. Impersonation attacks are a significant threat because adversarial perturbations allow attackers to disguise themselves as legitimate users. This can lead to severe consequences, including data breaches, system damage, and misuse of resources. However, research on such attacks in FR remains limited. Existing adversarial patch generation methods exhibit limited efficacy in impersonation attacks due to (1) the need for high attacker capabilities, (2) low attack success rates, and (3) excessive query requirements. To address these challenges, we propose a novel method SAP-DIFF that leverages diffusion models to generate adversarial patches via semantic perturbations in the latent space rather than direct pixel manipulation. We introduce an attention disruption mechanism to generate features unrelated to the original face, facilitating the creation of adversarial samples and a directional loss function to guide perturbations toward the target identity feature space, thereby enhancing attack effectiveness and efficiency. Extensive experiments on popular FR models and datasets demonstrate that our method outperforms state-of-the-art approaches, achieving an average attack success rate improvement of 45.66% (all exceeding 40%), and a reduction in the number of queries by about 40% compared to the SOTA approach
Abstract (translated)
为了评估面部识别（FR）模型的鲁棒性，许多研究集中在通过引入局部扰动误导FR模型的对抗补丁攻击上。冒充攻击是一种重大威胁，因为对抗性扰动允许攻击者伪装成合法用户，这可能导致数据泄露、系统损坏和资源滥用等严重后果。然而，在面部识别领域对这种攻击的研究仍然有限。现有的对抗补丁生成方法在实施冒充攻击时效果有限，原因在于：（1）需要较高的攻击能力；（2）较低的攻击成功率；以及（3）过高的查询需求。为解决这些挑战，我们提出了一种名为SAP-DIFF的新方法，该方法利用扩散模型通过潜在空间中的语义扰动而非直接像素操作生成对抗补丁。引入了注意力干扰机制来生成与原始面部无关的特征，从而促进对抗样本的创建，并采用方向性损失函数引导扰动朝向目标身份特征空间，以增强攻击的有效性和效率。在流行的FR模型和数据集上的大量实验表明，我们的方法优于现有最先进（SOTA）的方法，在所有情况下平均提高了45.66%的成功率（均超过40%），并且与SOTA方法相比查询次数减少了约40%。
URL
https://arxiv.org/abs/2502.19710
PDF
-
A Rapid Test for Accuracy and Bias of Face Recognition Technology
2025-02-20 19:38:52Manuel Knott, Ignacio Serna, Ethan Mann, Pietro PeronaAbstract
Measuring the accuracy of face recognition (FR) systems is essential for improving performance and ensuring responsible use. Accuracy is typically estimated using large annotated datasets, which are costly and difficult to obtain. We propose a novel method for 1:1 face verification that benchmarks FR systems quickly and without manual annotation, starting from approximate labels (e.g., from web search results). Unlike previous methods for training set label cleaning, ours leverages the embedding representation of the models being evaluated, achieving high accuracy in smaller-sized test datasets. Our approach reliably estimates FR accuracy and ranking, significantly reducing the time and cost of manual labeling. We also introduce the first public benchmark of five FR cloud services, revealing demographic biases, particularly lower accuracy for Asian women. Our rapid test method can democratize FR testing, promoting scrutiny and responsible use of the technology. Our method is provided as a publicly accessible tool at this https URL
Abstract (translated)
人脸识别（FR）系统准确性的测量对于提升性能和确保负责任的使用至关重要。准确性通常通过大型标注数据集来估计，而这些数据集的成本高昂且难以获取。我们提出了一种新颖的方法，用于1:1面部验证，该方法可以在不进行人工标注的情况下快速地对FR系统的性能进行基准测试，从近似的标签（例如来自网络搜索结果的标签）开始。与以前用于训练集标签清洗的方法不同，我们的方法利用了被评估模型的嵌入表示，在较小规模的数据集中实现了高准确性。 我们这种方法可靠地估计了FR系统的准确性和排名，显著减少了人工标注的时间和成本。此外，我们还推出了首个五种云服务人脸识别技术的公开基准测试，揭示了这些系统中存在的种族偏见，尤其是对亚洲女性的识别精度较低。 我们的快速测试方法可以促进更多人参与人脸验证技术的测试，并鼓励对该技术进行仔细审查和负责任地使用。这种方法已作为公共可用工具在以下链接提供：[此URL]（请将这里的URL替换为实际提供的链接）。
URL
https://arxiv.org/abs/2502.14996
PDF
-
A Comparison of Human and Machine Learning Errors in Face Recognition
2025-02-17 01:27:35Marina Est\'evez-Almenzar, Ricardo Baeza-Yates, Carlos CastilloAbstract
Machine learning applications in high-stakes scenarios should always operate under human oversight. Developing an optimal combination of human and machine intelligence requires an understanding of their complementarities, particularly regarding the similarities and differences in the way they make mistakes. We perform extensive experiments in the area of face recognition and compare two automated face recognition systems against human annotators through a demographically balanced user study. Our research uncovers important ways in which machine learning errors and human errors differ from each other, and suggests potential strategies in which human-machine collaboration can improve accuracy in face recognition.
Abstract (translated)
在高风险场景中应用机器学习时，系统应始终处于人类监督之下。为了开发出最佳的人机智能组合，需要理解它们各自的互补性，特别是关于他们在犯错方式上的相似性和差异性。我们通过一项人口学平衡的用户研究，在面部识别领域进行了广泛的实验，并将两个自动化的面部识别系统与人工标注员进行比较。我们的研究表明机器学习错误和人类错误之间存在重要的不同之处，并提出了可能的人机协作策略，以提高面部识别的准确性。
URL
https://arxiv.org/abs/2502.11337
PDF
-
Efficient Global Neural Architecture Search
2025-02-05 19:10:17Shahid Siddiqui, Christos Kyrkou, Theocharis TheocharidesAbstract
Neural architecture search (NAS) has shown promise towards automating neural network design for a given task, but it is computationally demanding due to training costs associated with evaluating a large number of architectures to find the optimal one. To speed up NAS, recent works limit the search to network building blocks (modular search) instead of searching the entire architecture (global search), approximate candidates' performance evaluation in lieu of complete training, and use gradient descent rather than naturally suitable discrete optimization approaches. However, modular search does not determine network's macro architecture i.e. depth and width, demanding manual trial and error post-search, hence lacking automation. In this work, we revisit NAS and design a navigable, yet architecturally diverse, macro-micro search space. In addition, to determine relative rankings of candidates, existing methods employ consistent approximations across entire search spaces, whereas different networks may not be fairly comparable under one training protocol. Hence, we propose an architecture-aware approximation with variable training schemes for different networks. Moreover, we develop an efficient search strategy by disjoining macro-micro network design that yields competitive architectures in terms of both accuracy and size. Our proposed framework achieves a new state-of-the-art on EMNIST and KMNIST, while being highly competitive on the CIFAR-10, CIFAR-100, and FashionMNIST datasets and being 2-4x faster than the fastest global search methods. Lastly, we demonstrate the transferability of our framework to real-world computer vision problems by discovering competitive architectures for face recognition applications.
Abstract (translated)
神经架构搜索（NAS）在自动化特定任务的神经网络设计方面显示出潜力，但由于评估大量架构以找到最优架构时涉及高昂的训练成本，这一过程计算密集。为了加速NAS，最近的工作将搜索限制在网络构建模块上（模块化搜索），而不是在整个架构中进行搜索（全局搜索），并且通过近似候选者的性能评价来替代完整训练，并使用梯度下降而非更适合的离散优化方法。然而，模块化搜索无法确定网络的整体结构即深度和宽度，在搜索后仍需手动试验调整，从而失去了自动化特性。 在这项工作中，我们重新审视了NAS设计了一个可导航且架构多样的宏-微搜索空间。此外，现有方法为了确定候选者的相对排名会在整个搜索空间中使用一致的近似法，但不同的网络在相同的训练协议下可能无法公平比较。因此，我们提出了基于架构感知的不同网络采用不同训练方案的近似策略。此外，通过分离宏-微网络设计，我们开发了一种高效的搜索策略，在准确性和大小方面都产生了竞争性的架构。我们的框架在EMNIST和KMNIST数据集上达到了新的SOTA（State of the Art），同时在CIFAR-10、CIFAR-100以及FashionMNIST数据集中表现出高度竞争力，并且比最快的全局搜索方法快2到4倍。 最后，我们通过为面部识别应用发现竞争性架构展示了框架向现实世界计算机视觉问题的可转移能力。
URL
https://arxiv.org/abs/2502.03553
PDF
-
Review of Demographic Bias in Face Recognition
2025-02-04 13:28:49Ketan Kotwal, Sebastien MarcelAbstract
Demographic bias in face recognition (FR) has emerged as a critical area of research, given its impact on fairness, equity, and reliability across diverse applications. As FR technologies are increasingly deployed globally, disparities in performance across demographic groups -- such as race, ethnicity, and gender -- have garnered significant attention. These biases not only compromise the credibility of FR systems but also raise ethical concerns, especially when these technologies are employed in sensitive domains. This review consolidates extensive research efforts providing a comprehensive overview of the multifaceted aspects of demographic bias in FR. We systematically examine the primary causes, datasets, assessment metrics, and mitigation approaches associated with demographic disparities in FR. By categorizing key contributions in these areas, this work provides a structured approach to understanding and addressing the complexity of this issue. Finally, we highlight current advancements and identify emerging challenges that need further investigation. This article aims to provide researchers with a unified perspective on the state-of-the-art while emphasizing the critical need for equitable and trustworthy FR systems.
Abstract (translated)
面部识别（FR）中的人口统计偏差已成为一个重要的研究领域，因为它对各种应用的公平性、公正性和可靠性产生了影响。随着面部识别技术在全球范围内的广泛应用，这些技术在不同的人口群体——如种族、民族和性别——之间的性能差异引起了广泛关注。这些偏见不仅损害了面部识别系统的可信度，还引发了伦理问题，特别是在这些技术被用于敏感领域时。本文综述了广泛的调研成果，提供了对面部识别中人口统计偏差的多方面概述。我们系统地审视了导致这种人口群体间差异的主要原因、数据集、评估指标和缓解方法。通过在这些领域的关键贡献进行分类，这项工作提供了一种结构化的方法来理解和解决这一问题的复杂性。最后，我们强调了当前的进步，并识别出需要进一步调查的新兴挑战。本文旨在为研究人员提供一个统一的观点，了解该领域最新的研究成果，并强调开发公平且值得信赖的人脸识别系统的重要性。
URL
https://arxiv.org/abs/2502.02309
PDF
-
UDC-VIT: A Real-World Video Dataset for Under-Display Cameras
2025-01-30 18:13:29Kyusu Ahn, JiSoo Kim, Sangik Lee, HyunGyu Lee, Byeonghyun Ko, Chanwoo Park, Jaejin LeeAbstract
Under Display Camera (UDC) is an advanced imaging system that places a digital camera lens underneath a display panel, effectively concealing the camera. However, the display panel significantly degrades captured images or videos, introducing low transmittance, blur, noise, and flare issues. Tackling such issues is challenging because of the complex degradation of UDCs, including diverse flare patterns. Despite extensive research on UDC images and their restoration models, studies on videos have yet to be significantly explored. While two UDC video datasets exist, they primarily focus on unrealistic or synthetic UDC degradation rather than real-world UDC degradation. In this paper, we propose a real-world UDC video dataset called UDC-VIT. Unlike existing datasets, only UDC-VIT exclusively includes human motions that target facial recognition. We propose a video-capturing system to simultaneously acquire non-degraded and UDC-degraded videos of the same scene. Then, we align a pair of captured videos frame by frame, using discrete Fourier transform (DFT). We compare UDC-VIT with six representative UDC still image datasets and two existing UDC video datasets. Using six deep-learning models, we compare UDC-VIT and an existing synthetic UDC video dataset. The results indicate the ineffectiveness of models trained on earlier synthetic UDC video datasets, as they do not reflect the actual characteristics of UDC-degraded videos. We also demonstrate the importance of effective UDC restoration by evaluating face recognition accuracy concerning PSNR, SSIM, and LPIPS scores. UDC-VIT enables further exploration in the UDC video restoration and offers better insights into the challenge. UDC-VIT is available at our project site.
Abstract (translated)
隐藏在显示屏下的摄像头（UDC）是一种高级成像系统，它将数字相机镜头放置于显示面板下方，从而有效隐藏了摄像头。然而，这种设计显著降低了拍摄图像或视频的质量，引入了透光率低、模糊度高、噪声和眩光等问题。解决这些问题极具挑战性，因为UDC的退化情况复杂多变，包括各种不同的眩光模式。尽管针对UDC图像及其恢复模型的研究已经非常广泛，但对于视频的相关研究却尚未得到充分探索。目前虽然存在两个UDC视频数据集，但它们主要关注的是不现实或合成的UDC退化问题，而非真实世界中的UDC退化。 在本文中，我们提出了一个名为UDC-VIT的真实世界UDC视频数据集。与现有的数据集不同，只有UDC-VIT专门包含了针对面部识别的人类动作场景。为了构建这一数据集，我们设计了一套视频采集系统，用于同时获取同一场景下的未退化和UDC退化的视频片段。然后，使用离散傅里叶变换（DFT），我们将捕获的一对视频逐帧对齐。 与六个具有代表性的UDC静态图像数据集以及两个现有的UDC视频数据集进行对比后，我们利用六种深度学习模型比较了UDC-VIT和现有合成UDC视频数据集。研究结果表明，基于早期合成UDC视频数据集训练的模型在处理真实世界中的UDC退化问题上效果不佳，因为它们未能准确反映实际的特征。 此外，通过评估PSNR、SSIM和LPIPS评分下的面部识别精度，我们展示了有效恢复UDC的重要性。UDC-VIT为UDC视频修复的研究提供了进一步探索的可能性，并更好地揭示了这一挑战的真实情况。UDC-VIT在我们的项目网站上可以获取。
URL
https://arxiv.org/abs/2501.18545
PDF
-
Frequency Matters: Explaining Biases of Face Recognition in the Frequency Domain
2025-01-28 12:27:25Marco Huber, Fadi Boutros, Naser DamerAbstract
Face recognition (FR) models are vulnerable to performance variations across demographic groups. The causes for these performance differences are unclear due to the highly complex deep learning-based structure of face recognition models. Several works aimed at exploring possible roots of gender and ethnicity bias, identifying semantic reasons such as hairstyle, make-up, or facial hair as possible sources. Motivated by recent discoveries of the importance of frequency patterns in convolutional neural networks, we explain bias in face recognition using state-of-the-art frequency-based explanations. Our extensive results show that different frequencies are important to FR models depending on the ethnicity of the samples.
Abstract (translated)
面部识别（FR）模型在不同的人口群体间表现存在差异，这些性能差距的原因尚不清楚，主要是因为基于深度学习的面部识别模型结构复杂。一些研究试图探索性别和种族偏见的根源，并发现诸如发型、妆容或胡须等语义因素可能是潜在原因。鉴于最近关于卷积神经网络中频率模式重要性的新发现，我们使用先进的基于频率的解释方法来解析面部识别中的偏见问题。我们的研究成果表明，不同的人种样本在面部识别模型中重要的频段是不同的。
URL
https://arxiv.org/abs/2501.16896
PDF
-
Device-aware Optical Adversarial Attack for a Portable Projector-camera System
2025-01-23 13:55:23Ning Jiang (School of Software & Microelectronics, Peking University, Beijing, China, Mashang Consumer Finance Co., Ltd., Chongqing, China), Yanhong Liu (Mashang Consumer Finance Co., Ltd., Chongqing, China), Dingheng Zeng (Mashang Consumer Finance Co., Ltd., Chongqing, China), Yue Feng (Mashang Consumer Finance Co., Ltd., Chongqing, China), Weihong Deng (Mashang Consumer Finance Co., Ltd., Chongqing, China), Ying Li (School of Software & Microelectronics, Peking University, Beijing, China)Abstract
Deep-learning-based face recognition (FR) systems are susceptible to adversarial examples in both digital and physical domains. Physical attacks present a greater threat to deployed systems as adversaries can easily access the input channel, allowing them to provide malicious inputs to impersonate a victim. This paper addresses the limitations of existing projector-camera-based adversarial light attacks in practical FR setups. By incorporating device-aware adaptations into the digital attack algorithm, such as resolution-aware and color-aware adjustments, we mitigate the degradation from digital to physical domains. Experimental validation showcases the efficacy of our proposed algorithm against real and spoof adversaries, achieving high physical similarity scores in FR models and state-of-the-art commercial systems. On average, there is only a 14% reduction in scores from digital to physical attacks, with high attack success rate in both white- and black-box scenarios.
Abstract (translated)
基于深度学习的人脸识别（FR）系统在数字和物理领域都容易受到对抗性样本的攻击。物理攻击对实际部署的系统构成了更大的威胁，因为对手可以轻松访问输入通道，从而提供恶意输入来冒充受害者。本文针对现有投影仪-摄像头基础的对抗光攻击在实际人脸识别设置中的局限性进行了探讨。通过将设备感知适应（如分辨率感知和颜色感知调整）融入数字攻击算法中，我们缓解了从数字到物理领域的质量下降问题。实验验证显示，我们的方法对真实和仿冒对手都具有高效率，在人脸识别模型及最先进的商用系统上均取得了高度的物理相似性得分。平均而言，从数字攻击到物理攻击的分数仅减少了14%，并且在白盒和黑盒场景中均实现了高成功率。
URL
https://arxiv.org/abs/2501.14005
PDF
-
LVFace: Large Vision model for Face Recogniton
2025-01-23 06:48:48Jinghan You, Yuanrui Sun, Mingyu Guo, Chao Feng, Jiao RanAbstract
Recently, large vision models have demonstrated powerful representation capabilities in the field of computer vision. However, we unexpectedly found that face recognition research is still mainly focused on CNN-based model architectures, which may lead to suboptimal state-of-the-art (SOTA) performance in face recognition. Therefore, we study how to use various loss functions from historical research orthogonally to train a new state-of-the-art face recognition model based on large vision models, called LVFace. On the largest public face database, WebFace42M, we demonstrated the superiority of LVFace over other advanced face recognition methods and achieved first place in the ICCV21 MFR-Ongoing challenge, until the submission of this work (December 30, 2024, academic track).
Abstract (translated)
最近，大型视觉模型在计算机视觉领域展示了强大的表示能力。然而，我们意外地发现面部识别研究仍然主要集中在基于CNN的模型架构上，这可能导致面部识别领域的最先进（SOTA）性能不佳。因此，我们研究了如何利用历史研究中的各种损失函数来正交训练一种基于大型视觉模型的新一代最先进面部识别模型，称为LVFace。在最大的公共人脸数据库WebFace42M上，我们展示了LVFace优于其他先进的面部识别方法，并在ICCV21 MFR-Ongoing挑战赛中获得第一名（截至提交此项工作时，即2024年12月30日的学术赛道）。
URL
https://arxiv.org/abs/2501.13420
PDF
-
On the 'Illusion' of Gender Bias in Face Recognition: Explaining the Fairness Issue Through Non-demographic Attributes
2025-01-21 10:21:19Paul Jonas Kurz, Haiyu Wu, Kevin W. Bowyer, Philipp Terh\"orstAbstract
Face recognition systems (FRS) exhibit significant accuracy differences based on the user's gender. Since such a gender gap reduces the trustworthiness of FRS, more recent efforts have tried to find the causes. However, these studies make use of manually selected, correlated, and small-sized sets of facial features to support their claims. In this work, we analyse gender bias in face recognition by successfully extending the search domain to decorrelated combinations of 40 non-demographic facial characteristics. First, we propose a toolchain to effectively decorrelate and aggregate facial attributes to enable a less-biased gender analysis on large-scale data. Second, we introduce two new fairness metrics to measure fairness with and without context. Based on these grounds, we thirdly present a novel unsupervised algorithm able to reliably identify attribute combinations that lead to vanishing bias when used as filter predicates for balanced testing datasets. The experiments show that the gender gap vanishes when images of male and female subjects share specific attributes, clearly indicating that the issue is not a question of biology but of the social definition of appearance. These findings could reshape our understanding of fairness in face biometrics and provide insights into FRS, helping to address gender bias issues.
Abstract (translated)
面部识别系统（FRS）在不同性别用户之间的准确度存在显著差异。这种性别差距降低了FRS的可信度，因此最近的研究努力尝试找出原因。然而，这些研究使用了手动挑选的小规模且相关联的人脸特征集来支持其论点。在这项工作中，我们通过将搜索领域扩展到40种非人口统计学面部特征的相关组合分析了面部识别中的性别偏见。 首先，我们提出了一套工具链，能够有效地去相关和聚合面部属性，以实现对大规模数据的较少偏向性性别分析。其次，我们引入了两个新的公平度量指标来衡量有无上下文情况下的公平性。基于这些基础，第三，我们提出了一个新颖的无监督算法，该算法可以可靠地识别出在平衡测试数据集中用作过滤条件时导致偏见消失的属性组合。 实验表明，当男性和女性的照片具有特定共同特征时，性别差距就会消失，这清楚地表明问题不在于生物学上的差异，而在于外貌的社会定义。这些发现可能会重塑我们对面部生物识别公平性的理解，并为解决面部识别系统中的性别偏见提供见解。
URL
https://arxiv.org/abs/2501.12020
PDF
-
Enhancing Sample Utilization in Noise-Robust Deep Metric Learning With Subgroup-Based Positive-Pair Selection
2025-01-19 14:41:55Zhipeng Yu, Qianqian Xu, Yangbangyan Jiang, Yingfei Sun, Qingming HuangAbstract
The existence of noisy labels in real-world data negatively impacts the performance of deep learning models. Although much research effort has been devoted to improving the robustness towards noisy labels in classification tasks, the problem of noisy labels in deep metric learning (DML) remains under-explored. Existing noisy label learning methods designed for DML mainly discard suspicious noisy samples, resulting in a waste of the training data. To address this issue, we propose a noise-robust DML framework with SubGroup-based Positive-pair Selection (SGPS), which constructs reliable positive pairs for noisy samples to enhance the sample utilization. Specifically, SGPS first effectively identifies clean and noisy samples by a probability-based clean sample selectionstrategy. To further utilize the remaining noisy samples, we discover their potential similar samples based on the subgroup information given by a subgroup generation module and then aggregate them into informative positive prototypes for each noisy sample via a positive prototype generation module. Afterward, a new contrastive loss is tailored for the noisy samples with their selected positive pairs. SGPS can be easily integrated into the training process of existing pair-wise DML tasks, like image retrieval and face recognition. Extensive experiments on multiple synthetic and real-world large-scale label noise datasets demonstrate the effectiveness of our proposed method. Without any bells and whistles, our SGPS framework outperforms the state-of-the-art noisy label DML methods. Code is available at \url{this https URL}.
Abstract (translated)
真实世界数据中存在的嘈杂标签会对深度学习模型的性能产生负面影响。尽管在提高分类任务对噪声标签鲁棒性方面投入了大量的研究努力，但关于深度度量学习（DML）中的噪声标签问题仍缺乏深入探索。现有的针对DML设计的噪声标签学习方法主要通过丢弃可疑的噪声样本来处理，导致了训练数据的浪费。为了解决这一问题，我们提出了一种基于子组正对选择（SGPS）的抗噪鲁棒深度度量学习框架，该框架构建了可靠的正对以增强噪声样本的使用效率。具体来说，SGPS首先通过一种概率基础的干净样本选择策略有效地识别干净和噪声样本。为进一步利用剩余的噪声样本，我们基于子组生成模块提供的子组信息发现其潜在相似样本，并通过正原型生成模块将其聚合为每个噪声样本的信息丰富正原型。随后，我们为带有选定正对的噪声样本定制了一种新的对比损失函数。SGPS可以轻松集成到现有的成对DML任务（如图像检索和人脸识别）训练过程中。在多个合成和真实世界的大规模标签噪声数据集上的广泛实验表明了我们提出的方法的有效性。无需任何复杂调整，我们的SGPS框架就超过了当前最先进的噪声标签DML方法的表现。代码可在[\url{this https URL}]获取。
URL
https://arxiv.org/abs/2501.11063
PDF
-
Generalized Single-Image-Based Morphing Attack Detection Using Deep Representations from Vision Transformer
2025-01-16 20:09:19Haoyu Zhang, Raghavendra Ramachandra, Kiran Raja, Christoph BuschAbstract
Face morphing attacks have posed severe threats to Face Recognition Systems (FRS), which are operated in border control and passport issuance use cases. Correspondingly, morphing attack detection algorithms (MAD) are needed to defend against such attacks. MAD approaches must be robust enough to handle unknown attacks in an open-set scenario where attacks can originate from various morphing generation algorithms, post-processing and the diversity of printers/scanners. The problem of generalization is further pronounced when the detection has to be made on a single suspected image. In this paper, we propose a generalized single-image-based MAD (S-MAD) algorithm by learning the encoding from Vision Transformer (ViT) architecture. Compared to CNN-based architectures, ViT model has the advantage on integrating local and global information and hence can be suitable to detect the morphing traces widely distributed among the face region. Extensive experiments are carried out on face morphing datasets generated using publicly available FRGC face datasets. Several state-of-the-art (SOTA) MAD algorithms, including representative ones that have been publicly evaluated, have been selected and benchmarked with our ViT-based approach. Obtained results demonstrate the improved detection performance of the proposed S-MAD method on inter-dataset testing (when different data is used for training and testing) and comparable performance on intra-dataset testing (when the same data is used for training and testing) experimental protocol.
Abstract (translated)
面部变形攻击已经对应用于边境控制和护照发放等场景的面部识别系统（FRS）构成了严重威胁。为此，需要开发能够防御此类攻击的面部变形检测算法（MAD）。这些算法必须足够稳健以处理开放集场景中的未知攻击，其中攻击可能源自多种不同的生成方法、后处理过程以及打印机/扫描仪的多样性。当仅基于一张疑似图像进行判断时，泛化问题更为突出。 本文提出了一种基于单张图片的通用面部变形检测（S-MAD）算法，并通过从视觉变压器（ViT）架构中学习编码来实现这一目标。与基于CNN的架构相比，ViT模型在整合局部和全局信息方面具有优势，因此更适于检测广泛分布在脸部区域中的变形痕迹。 我们在使用公开可用的FRGC面部数据集生成的脸部变形数据集上进行了广泛的实验，并将包括已公开评估在内的几种最先进的（SOTA）MAD算法与我们的基于ViT的方法进行了基准测试。结果表明，在跨数据集测试（即训练和测试使用的数据不同）时，所提出的S-MAD方法表现出改进的检测性能；在同数据集测试（即使用相同的数据进行训练和测试）时，则具有相当的检测性能。
URL
https://arxiv.org/abs/2501.09817
PDF
