Although face recognition systems have undergone an impressive evolution in the last decade, these technologies are vulnerable to attack presentations (AP). These attacks are mostly easy to create and, by executing them against the system's capture device, the malicious actor can impersonate an authorised subject and thus gain access to the latter's information (e.g., financial transactions). To protect facial recognition schemes against presentation attacks, state-of-the-art deep learning presentation attack detection (PAD) approaches require a large amount of data to produce reliable detection performances and even then, they decrease their performance for unknown presentation attack instruments (PAI) or database (information not seen during training), i.e. they lack generalisability. To mitigate the above problems, this paper focuses on zero-shot PAD. To do so, we first assess the effectiveness and generalisability of foundation models in established and challenging experimental scenarios and then propose a simple but effective framework for zero-shot PAD. Experimental results show that these models are able to achieve performance in difficult scenarios with minimal effort of the more advanced PAD mechanisms, whose weights were optimised mainly with training sets that included APs and bona fide presentations. The top-performing foundation model outperforms by a margin the best from the state of the art observed with the leaving-one-out protocol on the SiW-Mv2 database, which contains challenging unknown 2D and 3D attacks
尽管面部识别系统在过去的十年里经历了令人印象深刻的发展,但这些技术仍然容易受到攻击演示(AP)的影响。这些攻击通常是易于创建的,并且通过执行针对系统的捕捉设备,恶意行为者可以冒充授权用户并因此获得后者的个人信息(例如财务交易)。为了保护面部识别方案免受展示攻击,最先进的深度学习呈现攻击检测(PAD)方法需要大量的数据才能产生可靠的检测性能;即便如此,在面对未知呈现攻击工具(PAI)或未在训练期间见过的数据库信息时,它们的表现会下降,即缺乏泛化能力。为了解决上述问题,本文专注于零样本PAD。为此,我们首先评估了基础模型在已建立和具有挑战性的实验场景中的有效性和泛化性,并提出了一种简单的但有效的零样本PAD框架。实验结果表明,这些模型能够在困难的场景中以最少的努力实现高级PAD机制所达到的性能,而后者的主要训练集包括APs和真实的面部呈现数据。顶级表现的基础模型在SiW-Mv2数据库(包含具有挑战性的未知2D和3D攻击)上采用leave-one-out协议时的表现优于当前最佳方法。
https://arxiv.org/abs/2507.16393
Face image quality plays a critical role in determining the accuracy and reliability of face verification systems, particularly in real-time screening applications such as surveillance, identity verification, and access control. Low-quality face images, often caused by factors such as motion blur, poor lighting conditions, occlusions, and extreme pose variations, significantly degrade the performance of face recognition models, leading to higher false rejection and false acceptance rates. In this work, we propose a lightweight yet effective framework for automatic face quality assessment, which aims to pre-filter low-quality face images before they are passed to the verification pipeline. Our approach utilises normalised facial landmarks in conjunction with a Random Forest Regression classifier to assess image quality, achieving an accuracy of 96.67\%. By integrating this quality assessment module into the face verification process, we observe a substantial improvement in performance, including a comfortable 99.7\% reduction in the false rejection rate and enhanced cosine similarity scores when paired with the ArcFace face verification model. To validate our approach, we have conducted experiments on a real-world dataset collected comprising over 600 subjects captured from CCTV footage in unconstrained environments within Dubai Police. Our results demonstrate that the proposed framework effectively mitigates the impact of poor-quality face images, outperforming existing face quality assessment techniques while maintaining computational efficiency. Moreover, the framework specifically addresses two critical challenges in real-time screening: variations in face resolution and pose deviations, both of which are prevalent in practical surveillance scenarios.
面部图像质量在决定人脸识别系统的准确性和可靠性方面起着关键作用,尤其是在监控、身份验证和访问控制等实时筛选应用中。低质量的面部图像通常由运动模糊、光照条件差、遮挡以及极端姿态变化等因素引起,这会显著降低人脸识别模型的表现,导致更高的误拒率和误识率。在本研究中,我们提出了一种轻量级但有效的自动面部质量评估框架,旨在筛选出低质量的面部图像,然后再将其传递给身份验证流程。我们的方法利用标准化的面部特征点与随机森林回归分类器结合来评估图像质量,达到了96.67%的准确率。通过将这种质量评估模块集成到人脸识别过程中,我们观察到了显著的性能提升,包括误拒率下降了99.7%,并与ArcFace人脸识别模型搭配使用时,余弦相似度评分也有所提高。 为了验证我们的方法,我们在一个包含超过600名受试者的真实世界数据集上进行了实验,该数据集由迪拜警察在不受控环境中从闭路电视录像中采集。我们实验证明,提出的框架有效减轻了低质量面部图像的影响,在保持计算效率的同时超越了现有的面部质量评估技术。此外,本框架特别解决了实时筛选中的两个关键挑战:面部分辨率的变化和姿态的偏移,这两种情况在实际监控场景中普遍存在。
https://arxiv.org/abs/2507.15961
Recent advances in biometric systems have significantly improved the detection and prevention of fraudulent activities. However, as detection methods improve, attack techniques become increasingly sophisticated. Attacks on face recognition systems can be broadly divided into physical and digital approaches. Traditionally, deep learning models have been the primary defence against such attacks. While these models perform exceptionally well in scenarios for which they have been trained, they often struggle to adapt to different types of attacks or varying environmental conditions. These subsystems require substantial amounts of training data to achieve reliable performance, yet biometric data collection faces significant challenges, including privacy concerns and the logistical difficulties of capturing diverse attack scenarios under controlled conditions. This work investigates the application of Vision Language Models (VLM) and proposes an in-context learning framework for detecting physical presentation attacks and digital morphing attacks in biometric systems. Focusing on open-source models, the first systematic framework for the quantitative evaluation of VLMs in security-critical scenarios through in-context learning techniques is established. The experimental evaluation conducted on freely available databases demonstrates that the proposed subsystem achieves competitive performance for physical and digital attack detection, outperforming some of the traditional CNNs without resource-intensive training. The experimental results validate the proposed framework as a promising tool for improving generalisation in attack detection.
最近在生物识别系统方面的进展显著提高了检测和预防欺诈行为的能力。然而,随着检测方法的改进,攻击技术也变得越来越复杂。针对面部识别系统的攻击可以大致分为物理攻击和数字攻击两大类。传统上,深度学习模型一直是防御此类攻击的主要手段。尽管这些模型在其训练场景中表现出色,但它们往往难以适应不同类型或环境条件变化的攻击。为了实现可靠的性能,这类子系统需要大量的训练数据,而生物识别数据收集面临着包括隐私担忧及在控制条件下捕获各种攻击情景的实际困难等重大挑战。 本文研究了视觉语言模型(VLM)的应用,并提出了一种通过上下文学习框架检测生物识别系统中物理呈现攻击和数字变形攻击的方法。专注于开源模型,建立了首个基于上下文学习技术对安全关键场景中VLM进行定量评估的系统性框架。在公开数据库上进行的实验评估表明,所提出的子系统对于物理和数字攻击检测均实现了竞争性的性能,并且无需资源密集型训练即可超越某些传统的CNN(卷积神经网络)。这些实验结果验证了所提框架作为改进攻击检测泛化能力的有前景工具的有效性。
https://arxiv.org/abs/2507.15285
Adversarial attacks on face recognition systems (FRSs) pose serious security and privacy threats, especially when these systems are used for identity verification. In this paper, we propose a novel method for generating adversarial faces-synthetic facial images that are visually distinct yet recognized as a target identity by the FRS. Unlike iterative optimization-based approaches (e.g., gradient descent or other iterative solvers), our method leverages the structural characteristics of the FRS feature space. We figure out that individuals sharing the same attribute (e.g., gender or race) form an attributed subsphere. By utilizing such subspheres, our method achieves both non-adaptiveness and a remarkably small number of queries. This eliminates the need for relying on transferability and open-source surrogate models, which have been a typical strategy when repeated adaptive queries to commercial FRSs are impossible. Despite requiring only a single non-adaptive query consisting of 100 face images, our method achieves a high success rate of over 93% against AWS's CompareFaces API at its default threshold. Furthermore, unlike many existing attacks that perturb a given image, our method can deliberately produce adversarial faces that impersonate the target identity while exhibiting high-level attributes chosen by the adversary.
针对面部识别系统(FRS)的对抗性攻击会对安全和隐私构成严重威胁,尤其是在这些系统用于身份验证时。本文提出了一种生成对抗性面孔的新方法——即视觉上明显不同于目标个体但被面部识别系统识别为特定身份的合成面部图像。与基于迭代优化的方法(如梯度下降或其他迭代求解器)不同,我们的方法利用了FRS特征空间中的结构特性。我们发现,具有相同属性的人(例如性别或种族)形成了有向子球体。通过使用这些子球体,我们的方法既实现了非适应性又显著减少了查询次数。这消除了依赖于迁移性和开源替代模型的需求,后者通常是当无法对商用FRS进行重复适应性查询时的典型策略。尽管仅需一次包括100张面部图像的非适应性查询,我们的方法在针对AWS的CompareFaces API默认阈值下实现了超过93%的成功率。此外,与许多现有攻击通过扰动给定图像来操作不同的是,我们的方法可以故意生成具有由对手选择的高级别属性、冒充目标身份的对抗性面孔。
https://arxiv.org/abs/2507.12107
As face recognition systems (FRS) become more widely used, user privacy becomes more important. A key privacy issue in FRS is protecting the user's face template, as the characteristics of the user's face image can be recovered from the template. Although recent advances in cryptographic tools such as homomorphic encryption (HE) have provided opportunities for securing the FRS, HE cannot be used directly with FRS in an efficient plug-and-play manner. In particular, although HE is functionally complete for arbitrary programs, it is basically designed for algebraic operations on encrypted data of predetermined shape, such as a polynomial ring. Thus, a non-tailored combination of HE and the system can yield very inefficient performance, and many previous HE-based face template protection methods are hundreds of times slower than plain systems without protection. In this study, we propose IDFace, a new HE-based secure and efficient face identification method with template protection. IDFace is designed on the basis of two novel techniques for efficient searching on a (homomorphically encrypted) biometric database with an angular metric. The first technique is a template representation transformation that sharply reduces the unit cost for the matching test. The second is a space-efficient encoding that reduces wasted space from the encryption algorithm, thus saving the number of operations on encrypted templates. Through experiments, we show that IDFace can identify a face template from among a database of 1M encrypted templates in 126ms, showing only 2X overhead compared to the identification over plaintexts.
随着面部识别系统(FRS)的广泛应用,用户隐私变得越来越重要。在FRS中,一个关键的隐私问题是如何保护用户的面部模板,因为可以从该模板中恢复出用户面部图像的特征。尽管最近在密码学工具如同态加密(HE)方面取得了一些进展,为保障FRS的安全性提供了机会,但直接将HE应用于FRS并不能实现高效的一键式操作。特别是,虽然HE理论上可以用于任意程序的功能完成,并且主要是为了对预定形状(例如多项式环)的加密数据进行代数运算而设计,但在非定制组合中使用HE和系统时会显著降低效率,许多基于HE的方法在保护面部模板方面比没有保护的传统系统慢数百倍。 本文提出了一种新的、基于HE的安全高效的人脸识别方法——IDFace,并具有模板保护功能。该方法基于两种用于在生物特征数据库(同态加密)中以角度度量进行有效搜索的创新技术而设计。第一项技术是一种模板表示转换,它显著降低了匹配测试中的单次成本;第二项技术则是一种节省空间的编码方式,通过减少加密算法造成的浪费空间来降低对加密模板的操作次数。 实验表明,IDFace能够在126毫秒内从一个包含一百万个加密面部模板的数据库中识别出一个特定的面部模板,并且仅比处理明文数据时多花费两倍的时间。
https://arxiv.org/abs/2507.12050
Recently, personalized portrait generation with a text-to-image diffusion model has significantly advanced with Textual Inversion, emerging as a promising approach for creating high-fidelity personalized images. Despite its potential, current Textual Inversion methods struggle to maintain consistent facial identity due to semantic misalignments between textual and visual embedding spaces regarding identity. We introduce ID-EA, a novel framework that guides text embeddings to align with visual identity embeddings, thereby improving identity preservation in a personalized generation. ID-EA comprises two key components: the ID-driven Enhancer (ID-Enhancer) and the ID-conditioned Adapter (ID-Adapter). First, the ID-Enhancer integrates identity embeddings with a textual ID anchor, refining visual identity embeddings derived from a face recognition model using representative text embeddings. Then, the ID-Adapter leverages the identity-enhanced embedding to adapt the text condition, ensuring identity preservation by adjusting the cross-attention module in the pre-trained UNet model. This process encourages the text features to find the most related visual clues across the foreground snippets. Extensive quantitative and qualitative evaluations demonstrate that ID-EA substantially outperforms state-of-the-art methods in identity preservation metrics while achieving remarkable computational efficiency, generating personalized portraits approximately 15 times faster than existing approaches.
最近,使用文本到图像扩散模型进行个性化肖像生成在引入Textual Inversion技术后取得了显著进展,成为创建高保真个性化图像的一种有前景的方法。尽管具有巨大潜力,现有的Textual Inversion方法却难以保持面部身份的一致性,因为文本和视觉嵌入空间之间关于身份的语义不匹配导致了这个问题。我们提出了ID-EA(Identity-Driven Embedding Alignment),这是一种新的框架,它引导文本嵌入与视觉身份嵌入对齐,从而在个性化生成中提高身份保存的效果。 ID-EA包括两个关键组成部分:由身份驱动的增强器(ID-Enhancer)和根据身份条件适配器(ID-Adapter)。首先,ID-Enhancer 将身份嵌入与文本 ID 锚点集成起来,并使用来自面部识别模型的视觉身份嵌入对代表性文本嵌入进行细化。然后,ID-Adapter 利用增强后的身份嵌入来调整文本条件,在预训练的 UNet 模型中通过调节交叉注意力模块确保身份保存。这一过程鼓励文本特征找到前景片段中的最相关视觉线索。 广泛的定量和定性评估表明,与现有方法相比,ID-EA 在身份保持度量方面显著超越了最先进的技术,并且实现了令人瞩目的计算效率,在生成个性化肖像时比现有的方法快大约15倍。
https://arxiv.org/abs/2507.11990
Face Recognition (FR) tasks have made significant progress with the advent of Deep Neural Networks, particularly through margin-based triplet losses that embed facial images into high-dimensional feature spaces. During training, these contrastive losses focus exclusively on identity information as labels. However, we observe a multiscale geometric structure emerging in the embedding space, influenced by interpretable facial (e.g., hair color) and image attributes (e.g., contrast). We propose a geometric approach to describe the dependence or invariance of FR models to these attributes and introduce a physics-inspired alignment metric. We evaluate the proposed metric on controlled, simplified models and widely used FR models fine-tuned with synthetic data for targeted attribute augmentation. Our findings reveal that the models exhibit varying degrees of invariance across different attributes, providing insight into their strengths and weaknesses and enabling deeper interpretability. Code available here: this https URL}{this https URL
面部识别(FR)任务随着深度神经网络的出现取得了显著进展,特别是通过基于边距的三元组损失将面部图像嵌入到高维特征空间中。在训练过程中,这些对比损失仅关注身份信息作为标签。然而,我们观察到,在嵌入空间中出现了由可解释的面部(如发色)和图像属性(如对比度)影响的多尺度几何结构。我们提出了一种基于几何的方法来描述FR模型对这些属性的依赖或不变性,并引入了一个受物理学启发的对齐指标。我们在控制下的简化模型和广泛使用的FR模型上评估了该指标,后者使用合成数据进行了微调以增强特定属性。我们的研究结果表明,不同属性下模型表现出不同程度的不变性,这为理解它们的优势与劣势提供了洞见,并促进了更深层次的理解。代码可在以下链接获取:[此处插入具体URL]
https://arxiv.org/abs/2507.11372
This paper aims to propose a novel machine learning (ML) approach incorporating Homomorphic Encryption (HE) to address privacy limitations in Unmanned Aerial Vehicles (UAV)-based face detection. Due to challenges related to distance, altitude, and face orientation, high-resolution imagery and sophisticated neural networks enable accurate face recognition in dynamic environments. However, privacy concerns arise from the extensive surveillance capabilities of UAVs. To resolve this issue, we propose a novel framework that integrates HE with advanced neural networks to secure facial data throughout the inference phase. This method ensures that facial data remains secure with minimal impact on detection accuracy. Specifically, the proposed system leverages the Cheon-Kim-Kim-Song (CKKS) scheme to perform computations directly on encrypted data, optimizing computational efficiency and security. Furthermore, we develop an effective data encoding method specifically designed to preprocess the raw facial data into CKKS form in a Single-Instruction-Multiple-Data (SIMD) manner. Building on this, we design a secure inference algorithm to compute on ciphertext without needing decryption. This approach not only protects data privacy during the processing of facial data but also enhances the efficiency of UAV-based face detection systems. Experimental results demonstrate that our method effectively balances privacy protection and detection performance, making it a viable solution for UAV-based secure face detection. Significantly, our approach (while maintaining data confidentially with HE encryption) can still achieve an accuracy of less than 1% compared to the benchmark without using encryption.
本文旨在提出一种结合同态加密(HE)的新型机器学习(ML)方法,以解决无人机(UAV)面部检测中的隐私限制问题。由于距离、高度和面部方向的变化带来的挑战,高分辨率图像和复杂的神经网络能够在动态环境中实现准确的人脸识别。然而,UAV的强大监控能力引发了隐私担忧。为了解决这一问题,我们提出了一种新颖的框架,该框架将HE与高级神经网络相结合,在推理阶段保护面部数据的安全性。这种方法确保了面部数据在安全的前提下不会显著影响检测准确性。 具体而言,所提出的系统利用Cheon-Kim-Kim-Song(CKKS)方案直接对加密数据进行计算,从而优化了计算效率和安全性。此外,我们开发了一种有效的数据编码方法,该方法专门用于将原始面部数据预处理为适合CKKS的格式,并采用单指令多数据流(SIMD)方式进行处理。在此基础上,设计了一个安全推理算法,在不解密的情况下对加密后的数据进行计算。这种方法不仅在处理人脸数据时保护了数据隐私,还提高了基于UAV的人脸检测系统的效率。 实验结果表明,我们提出的方法能够有效平衡隐私保护与检测性能之间的关系,使其成为基于UAV的面部识别系统的一种可行解决方案。特别值得注意的是,在保持数据机密性的同时(通过HE加密),我们的方法相比不使用加密的标准基准仍能达到不到1%误差率的准确性。
https://arxiv.org/abs/2507.09860
With the increasing deployment of intelligent CCTV systems in outdoor environments, there is a growing demand for face recognition systems optimized for challenging weather conditions. Adverse weather significantly degrades image quality, which in turn reduces recognition accuracy. Although recent face image restoration (FIR) models based on generative adversarial networks (GANs) and diffusion models have shown progress, their performance remains limited due to the lack of dedicated modules that explicitly address weather-induced degradations. This leads to distorted facial textures and structures. To address these limitations, we propose a novel GAN-based blind FIR framework that integrates two key components: local Statistical Facial Feature Transformation (SFFT) and Degradation-Agnostic Feature Embedding (DAFE). The local SFFT module enhances facial structure and color fidelity by aligning the local statistical distributions of low-quality (LQ) facial regions with those of high-quality (HQ) counterparts. Complementarily, the DAFE module enables robust statistical facial feature extraction under adverse weather conditions by aligning LQ and HQ encoder representations, thereby making the restoration process adaptive to severe weather-induced degradations. Experimental results demonstrate that the proposed degradation-agnostic SFFT model outperforms existing state-of-the-art FIR methods based on GAN and diffusion models, particularly in suppressing texture distortions and accurately reconstructing facial structures. Furthermore, both the SFFT and DAFE modules are empirically validated in enhancing structural fidelity and perceptual quality in face restoration under challenging weather scenarios.
随着智能CCTV系统在户外环境中的部署日益增加,对于能够在恶劣天气条件下优化的面部识别系统的市场需求也在不断增长。不良天气条件显著降低图像质量,进而减少了识别准确率。尽管基于生成对抗网络(GAN)和扩散模型的最新面部图像恢复(FIR)方法显示出了一定的进步,但由于缺乏专门针对由天气引起的退化问题进行处理的模块,其性能仍然有限,导致了面部纹理和结构变形。 为了克服这些限制,我们提出了一种新颖的基于GAN的盲态FIR框架,该框架集成了两个关键组件:局部统计面部特征转换(SFFT)以及不受退化影响的特征嵌入(DAFE)。局部SFFT模块通过使低质量(LQ)面部区域与高质量(HQ)对应部分之间的局部统计分布对齐来增强面部结构和颜色保真度。同时,DAFE模块允许在恶劣天气条件下进行鲁棒性统计面部特征提取,通过使LQ和HQ编码器表示对齐,从而使恢复过程适应于严重的由天气引起的退化。 实验结果表明,所提出的不受退化影响的SFFT模型优于现有的基于GAN和扩散模型的FIR方法,在抑制纹理失真并准确重建面部结构方面尤其有效。此外,无论是SFFT模块还是DAFE模块都在挑战性天气场景下的面部恢复过程中被实证验证为能够增强结构保真度和感知质量。
https://arxiv.org/abs/2507.07464
This study presents findings from long-term biometric evaluations conducted at the Biometric Evaluation Center (bez). Over the course of two and a half years, our ongoing research with over 400 participants representing diverse ethnicities, genders, and age groups were regularly assessed using a variety of biometric tools and techniques at the controlled testing facilities. Our findings are based on the General Data Protection Regulation-compliant local bez database with more than 238.000 biometric data sets categorized into multiple biometric modalities such as face and finger. We used state-of-the-art face recognition algorithms to analyze long-term comparison scores. Our results show that these scores fluctuate more significantly between individual days than over the entire measurement period. These findings highlight the importance of testing biometric characteristics of the same individuals over a longer period of time in a controlled measurement environment and lays the groundwork for future advancements in biometric data analysis.
这项研究展示了在生物特征评估中心(bez)进行的长期生物识别评估的结果。历时两年半,我们持续的研究项目涉及超过400名参与者,这些参与者涵盖了不同的种族、性别和年龄群体,并且他们定期使用多种生物识别工具和技术,在受控测试设施内接受评估。 我们的发现基于符合《通用数据保护条例》的地方bez数据库,该数据库包含超过238,000份分类为面部、指纹等多种生物特征模式的生物识别数据集。我们采用了最先进的面部识别算法来分析长期比较得分,并且研究结果表明这些得分在不同的日子之间波动较大,但在整个测量期间相对稳定。 这些发现强调了在受控环境中长时间测试同一对象的生物特征的重要性,并为未来生物识别数据分析的进步奠定了基础。
https://arxiv.org/abs/2507.06858
In this paper, we address the following question: How do generic foundation models (e.g., CLIP, BLIP, LLaVa, DINO) compare against a domain-specific face recognition model (viz., AdaFace or ArcFace) on the face recognition task? Through a series of experiments involving several foundation models and benchmark datasets, we are able to report the following findings: (a) In all datasets considered, domain-specific models outperformed zero-shot foundation models. (b) The performance of zero-shot generic foundation models improves on over-segmented face images than tightly cropped faces thereby suggesting the importance of contextual clues. For example, at a False Match Rate (FMR) of 0.01%, the True Match Rate (TMR) of OpenCLIP improved from 64.97% to 81.73% on the LFW dataset as the face crop increased from 112x112 to 250x250 while the TMR of domain-specific AdaFace dropped from 99.09% to 77.31%. (c) A simple score-level fusion of a foundation model with a domain-specific FR model improved the accuracy at low FMRs. For example, the TMR of AdaFace when fused with BLIP improved from 72.64% to 83.31% at an FMR of 0.0001% on the IJB-B dataset and from 73.17% to 85.81% on the IJB-C dataset. (d) Foundation models, such as ChatGPT, can be used to impart explainability to the FR pipeline (e.g., ``Despite minor lighting and head tilt differences, the two left-profile images show high consistency in forehead slope, nose shape, chin contour...''). In some instances, foundation models are even able to resolve low-confidence decisions made by AdaFace (e.g., ``Although AdaFace assigns a low similarity score of 0.21, both images exhibit visual similarity...and the pair is likely of the same person''), thereby reiterating the importance of combining domain-specific FR models with generic foundation models in a judicious manner.
在这篇论文中,我们探讨了以下问题:通用基础模型(如CLIP、BLIP、LLaVa、DINO)与专用于人脸识别的领域特定模型(例如AdaFace或ArcFace)在人脸识别任务上的表现如何?通过一系列涉及多个基础模型和基准数据集的实验,我们得出了如下发现: (a) 在所有考虑的数据集中,领域特定模型的表现优于零样本基础模型。 (b) 零样本通用基础模型在处理过度分割的人脸图像时比紧密裁剪的人脸图像表现出更好的性能,这表明上下文线索的重要性。例如,在错误匹配率为0.01%的情况下,OpenCLIP在LFW数据集上的真匹配率从64.97%提升到了81.73%,当面部裁剪尺寸由112x112增加到250x250时;相比之下,领域特定的AdaFace模型的真匹配率却从99.09%下降到了77.31%。 (c) 将基础模型与专有的人脸识别模型进行简单的分数级融合可以提高低错误匹配率下的准确性。例如,在IJB-B数据集上,当AdaFace与BLIP融合后,以0.0001%的FMR计算时,真匹配率从72.64%提升到了83.31%,而在IJB-C数据集上的真匹配率则从73.17%提高到85.81%。 (d) 像ChatGPT这样的基础模型可以用于为人脸识别管道赋予解释性(例如,“尽管光线和头部角度略有不同,两张侧面照在前额斜度、鼻形和下巴轮廓方面表现出高度一致性……”)。在某些情况下,基础模型甚至能够解决AdaFace做出的低置信度决策(例如,“虽然AdaFace给出的相似分数为0.21较低,但两幅图像具有视觉上的相似性……这对可能是同一个人的照片。”),这进一步强调了以明智的方式结合领域特定的人脸识别模型与通用基础模型的重要性。
https://arxiv.org/abs/2507.03541
Masked Face Recognition (MFR) is an increasingly important area in biometric recognition technologies, especially with the widespread use of masks as a result of the COVID-19 pandemic. This development has created new challenges for facial recognition systems due to the partial concealment of basic facial features. This paper aims to provide a comprehensive review of the latest developments in the field, with a focus on deep learning techniques, especially convolutional neural networks (CNNs) and twin networks (Siamese networks), which have played a pivotal role in improving the accuracy of covering face recognition. The paper discusses the most prominent challenges, which include changes in lighting, different facial positions, partial concealment, and the impact of mask types on the performance of systems. It also reviews advanced technologies developed to overcome these challenges, including data enhancement using artificial databases and multimedia methods to improve the ability of systems to generalize. In addition, the paper highlights advance in deep network design, feature extraction techniques, evaluation criteria, and data sets used in this area. Moreover, it reviews the various applications of masked face recognition in the fields of security and medicine, highlighting the growing importance of these systems in light of recurrent health crises and increasing security threats. Finally, the paper focuses on future research trends such as developing more efficient algorithms and integrating multimedia technologies to improve the performance of recognition systems in real-world environments and expand their applications.
口罩面部识别(MFR)是生物识别技术领域中日益重要的一个分支,尤其是在新冠疫情导致广泛使用口罩的情况下。这一发展为面部识别系统带来了新的挑战,因为基本的面部特征部分被遮盖。本文旨在全面回顾该领域的最新进展,重点关注深度学习技术,尤其是卷积神经网络(CNNs)和孪生网络(Siamese networks),这些技术在提高覆盖脸孔识别精度方面发挥了关键作用。 论文讨论了最突出的挑战,包括光照变化、不同面部位置、部分遮挡以及口罩类型对系统性能的影响。此外,还回顾了为克服这些挑战而开发的先进技术,例如使用人工数据库进行数据增强和多媒体方法以提升系统的泛化能力。同时,文章强调了深度网络设计的进步、特征提取技术、评估标准及在此领域使用的数据集。 此外,论文审查了口罩面部识别在安全和医疗领域的各种应用,并突出了这些系统在面对反复出现的健康危机和日益严峻的安全威胁时的重要性。 最后,该文聚焦于未来的研究趋势,包括开发更高效的算法以及整合多媒体技术以提升现实环境中识别系统的性能并扩展其应用范围。
https://arxiv.org/abs/2508.00841
Face identification systems operating in the ciphertext domain have garnered significant attention due to increasing privacy concerns and the potential recovery of original facial data. However, as the size of ciphertext template libraries grows, the face retrieval process becomes progressively more time-intensive. To address this challenge, we propose a novel and efficient scheme for face retrieval in the ciphertext domain, termed Privacy-Preserving Preselection for Face Identification Based on Packing (PFIP). PFIP incorporates an innovative preselection mechanism to reduce computational overhead and a packing module to enhance the flexibility of biometric systems during the enrollment stage. Extensive experiments conducted on the LFW and CASIA datasets demonstrate that PFIP preserves the accuracy of the original face recognition model, achieving a 100% hit rate while retrieving 1,000 ciphertext face templates within 300 milliseconds. Compared to existing approaches, PFIP achieves a nearly 50x improvement in retrieval efficiency.
基于加密域中的人脸识别系统由于隐私保护需求和防止原始面部数据恢复的潜在风险而备受关注。然而,随着加密模板库大小的增长,人脸检索过程变得越来越耗时。为解决这一挑战,我们提出了一种新颖且高效的方案——基于包装(Packing)的身份认证隐私保护预选方案(PFIP),用于加密域中的面部识别。该方案集成了创新的预选择机制以减少计算开销,并引入了一个打包模块来增强生物特征系统在注册阶段的灵活性。 在LFW和CASIA数据集中进行的广泛实验表明,PFIP能够保持原始人脸识别模型的准确性,在检索1,000个加密人脸模板时实现100%命中率且耗时不超过300毫秒。与现有方法相比,PFIP在检索效率上实现了近50倍的提升。
https://arxiv.org/abs/2507.02414
The widespread use of deep learning face recognition raises several security concerns. Although prior works point at existing vulnerabilities, DNN backdoor attacks against real-life, unconstrained systems dealing with images captured in the wild remain a blind spot of the literature. This paper conducts the first system-level study of backdoors in deep learning-based face recognition systems. This paper yields four contributions by exploring the feasibility of DNN backdoors on these pipelines in a holistic fashion. We demonstrate for the first time two backdoor attacks on the face detection task: face generation and face landmark shift attacks. We then show that face feature extractors trained with large margin losses also fall victim to backdoor attacks. Combining our models, we then show using 20 possible pipeline configurations and 15 attack cases that a single backdoor enables an attacker to bypass the entire function of a system. Finally, we provide stakeholders with several best practices and countermeasures.
深度学习面部识别的广泛应用引发了几项安全问题。尽管先前的研究指出了现有的漏洞,但针对实际环境中捕获图像的真实无约束系统进行DNN后门攻击仍然是文献中的盲点。本文首次对基于深度学习的面部识别系统的后门进行了系统级研究。本文通过全面探索这些管道中DNN后门的可能性,在四个方面做出了贡献。我们首次展示了在面部检测任务上的两个后门攻击:面部生成和面部关键点偏移攻击。接着,我们还表明使用大边距损失训练的面部特征提取器同样容易受到后门攻击的影响。结合我们的模型,我们通过20种可能的管道配置和15个攻击案例展示了一个单一的后门就足以使攻击者绕过整个系统的功能。最后,本文为利益相关者提供了几种最佳实践和对策。
https://arxiv.org/abs/2507.01607
This study presents a novel classroom surveillance system that integrates multiple modalities, including drowsiness, tracking of mobile phone usage, and face recognition,to assess student attentiveness with enhanced this http URL system leverages the YOLOv8 model to detect both mobile phone and sleep usage,(Ghatge et al., 2024) while facial recognition is achieved through LResNet Occ FC body tracking using YOLO and MTCNN.(Durai et al., 2024) These models work in synergy to provide comprehensive, real-time monitoring, offering insights into student engagement and behavior.(S et al., 2023) The framework is trained on specialized datasets, such as the RMFD dataset for face recognition and a Roboflow dataset for mobile phone detection. The extensive evaluation of the system shows promising results. Sleep detection achieves 97. 42% mAP@50, face recognition achieves 86. 45% validation accuracy and mobile phone detection reach 85. 89% mAP@50. The system is implemented within a core PHP web application and utilizes ESP32-CAM hardware for seamless data capture.(Neto et al., 2024) This integrated approach not only enhances classroom monitoring, but also ensures automatic attendance recording via face recognition as students remain seated in the classroom, offering scalability for diverse educational environments.(Banada,2025)
这项研究提出了一种新型的课堂监控系统,该系统集成了多种模态,包括检测瞌睡、手机使用跟踪和面部识别等,以增强对学生注意力的评估。此系统的开发基于YOLOv8模型来检测手机使用和睡眠状态(Ghatge等人,2024年),同时通过LResNet Occ FC身体追踪结合YOLO和MTCNN实现了面部识别功能(Durai等人,2024年)。这些模型协同工作,提供了全面的实时监控,为了解学生参与度和行为模式提供深入了解。(S等,2023年) 该框架是在专门的数据集上进行训练的,例如用于面部识别的RMFD数据集和用于手机检测的Roboflow数据集。系统经过广泛的评估显示了令人鼓舞的结果:睡眠检测的平均精度(mAP@50)达到了97.42%,面部识别验证准确率为86.45%,而手机检测则达到85.89% mAP@50。 该系统的实现采用了一种核心PHP网络应用程序,并使用ESP32-CAM硬件来无缝采集数据。(Neto等人,2024年)这种综合方法不仅提高了课堂监控的效果,而且还通过面部识别自动记录学生的出勤情况,确保学生在座位上时即被系统捕捉到,从而为各种教育环境提供了可扩展性(Banada, 2025)。
https://arxiv.org/abs/2507.01590
In current practical face authentication systems, most face recognition (FR) algorithms are based on cosine similarity with softmax classification. Despite its reliable classification performance, this method struggles with hard samples. A popular strategy to improve FR performance is incorporating angular or cosine margins. However, it does not take face quality or recognition hardness into account, simply increasing the margin value and thus causing an overly uniform training strategy. To address this problem, a novel loss function is proposed, named Loss function for Hard High-quality Face (LH2Face). Firstly, a similarity measure based on the von Mises-Fisher (vMF) distribution is stated, specifically focusing on the logarithm of the Probability Density Function (PDF), which represents the distance between a probability distribution and a vector. Then, an adaptive margin-based multi-classification method using softmax, called the Uncertainty-Aware Margin Function, is implemented in the article. Furthermore, proxy-based loss functions are used to apply extra constraints between the proxy and sample to optimize their representation space distribution. Finally, a renderer is constructed that optimizes FR through face reconstruction and vice versa. Our LH2Face is superior to similiar schemes on hard high-quality face datasets, achieving 49.39% accuracy on the IJB-B dataset, which surpasses the second-place method by 2.37%.
在当前的实际面部认证系统中,大多数面部识别(FR)算法都是基于余弦相似度和softmax分类。尽管这种方法具有可靠的分类性能,但在处理难以识别的样本时表现不佳。一种常用的改进面部识别性能的方法是引入角度或余弦边距。然而,这种策略没有考虑面部质量或识别难度的问题,只是简单地增加边距值,从而导致了过于统一的训练策略。为了解决这个问题,提出了一种新的损失函数,名为“针对硬高质量人脸的损失函数”(Loss function for Hard High-quality Face, LH2Face)。 首先,该方法基于冯·米塞斯-费希尔(von Mises-Fisher, vMF)分布提出了一个相似度测量标准,并特别关注了概率密度函数(PDF)的对数值,这代表了一个概率分布与向量之间的距离。然后,在文章中实现了一种使用softmax的不确定性感知边距函数的多分类方法,这是一种自适应边距基的方法。此外,还采用了基于代理的损失函数来在代理和样本之间施加额外约束,以优化它们的表示空间分布。最后,构建了一个渲染器,通过面部重建和反向过程来进行面部识别的优化。 我们的LH2Face方案在硬高质量的人脸数据集上优于类似方案,在IJB-B数据集上的准确率为49.39%,比第二名的方法高出2.37%。
https://arxiv.org/abs/2506.23555
Burstiness, a phenomenon observed in text and image retrieval, refers to that particular elements appear more times in a set than a statistically independent model assumes. We argue that in the context of set-based face recognition (SFR), burstiness exists widely and degrades the performance in two aspects: Firstly, the bursty faces, where faces with particular attributes %exist frequently in a face set, dominate the training instances and dominate the training face sets and lead to poor generalization ability to unconstrained scenarios. Secondly, the bursty faces %dominating the evaluation sets interfere with the similarity comparison in set verification and identification when evaluation. To detect the bursty faces in a set, we propose three strategies based on Quickshift++, feature self-similarity, and generalized max-pooling (GMP). We apply the burst detection results on training and evaluation stages to enhance the sampling ratios or contributions of the infrequent faces. When evaluation, we additionally propose the quality-aware GMP that enables awareness of the face quality and robustness to the low-quality faces for the original GMP. We give illustrations and extensive experiments on the SFR benchmarks to demonstrate that burstiness is widespread and suppressing burstiness considerably improves the recognition performance.
突发性(Burstiness)是文本和图像检索中观察到的现象,指的是某些元素在集合中出现的次数比统计独立模型所假设的要多。我们提出,在基于集合的人脸识别(Set-Based Face Recognition, SFR)中,突发性普遍存在,并从两个方面影响性能:首先,具有特定属性的人脸频繁出现在人脸集中,主导了训练实例和训练集,导致在不受约束场景下的泛化能力较差;其次,在评估集中占主导地位的这些突发性人脸会干扰集合验证和识别过程中的相似度比较。 为了检测集合中突发性的面孔,我们提出了三种策略:基于Quickshift++、特征自相似性和广义最大池化(Generalized Max-Pooling, GMP)的方法。在训练和评估阶段应用突发检测结果以增加稀有面孔的采样比例或贡献值。此外,在评估过程中,我们还提出了一种质量感知GMP方法,使原始GMP能够识别面部质量并提高对低质量面部的鲁棒性。 我们在SFR基准测试上提供了详细的说明和广泛的实验来展示突发性的广泛存在,并表明抑制突发性可显著提升人脸识别性能。
https://arxiv.org/abs/2506.20312
Backdoor attacks embed a hidden functionality into deep neural networks, causing the network to display anomalous behavior when activated by a predetermined pattern in the input Trigger, while behaving well otherwise on public test data. Recent works have shown that backdoored face recognition (FR) systems can respond to natural-looking triggers like a particular pair of sunglasses. Such attacks pose a serious threat to the applicability of FR systems in high-security applications. We propose a novel technique to (1) detect whether an FR network is compromised with a natural, physically realizable trigger, and (2) identify such triggers given a compromised network. We demonstrate the effectiveness of our methods with a compromised FR network, where we are able to identify the trigger (e.g., green sunglasses or red hat) with a top-5 accuracy of 74%, whereas a naive brute force baseline achieves 56% accuracy.
后门攻击将隐藏功能嵌入深度神经网络,使得当输入触发器(Trigger)中包含预设模式时,该网络会表现出异常行为;而在其他情况下,则在网络的公开测试数据上表现良好。最近的研究表明,被植入后门的人脸识别系统会对一些自然存在的触发器做出响应,例如特定的一副太阳镜。此类攻击对人脸识别系统的高安全性应用构成了严重威胁。 我们提出了一种新的技术方法来: 1. 检测一个人脸识别网络是否被一种自然、可物理实现的触发器所侵入; 2. 在一个已经遭到入侵的人脸识别系统中,找出这样的触发器。 通过使用一个已被植入后门的人脸识别系统作为案例,我们能够以74%的准确率识别出具体的触发器(例如绿色太阳镜或红色帽子),而相比之下,一种简单的暴力破解基线方法只能达到56%的准确度。
https://arxiv.org/abs/2506.19533
Face identity provides a powerful signal for deepfake detection. Prior studies show that even when not explicitly modeled, classifiers often learn identity features implicitly. This has led to conflicting views: some suppress identity cues to reduce bias, while others rely on them as forensic evidence. To reconcile these views, we analyze two hypotheses: (1) whether face identity alone is discriminative for detecting deepfakes, and (2) whether such identity features generalize poorly across manipulation methods. Our experiments confirm that identity is informative but context-dependent. While some manipulations preserve identity-consistent artifacts, others distort identity cues and harm generalization. We argue that identity features should neither be blindly suppressed nor relied upon, but instead be explicitly modeled and adaptively controlled based on per-sample relevance. We propose \textbf{SELFI} (\textbf{SEL}ective \textbf{F}usion of \textbf{I}dentity), a generalizable detection framework that dynamically modulates identity usage. SELFI consists of: (1) a Forgery-Aware Identity Adapter (FAIA) that extracts identity embeddings from a frozen face recognition model and projects them into a forgery-relevant space via auxiliary supervision; and (2) an Identity-Aware Fusion Module (IAFM) that selectively integrates identity and visual features using a relevance-guided fusion mechanism. Experiments on four benchmarks show that SELFI improves cross-manipulation generalization, outperforming prior methods by an average of 3.1\% AUC. On the challenging DFDC dataset, SELFI exceeds the previous best by 6\%. Code will be released upon paper acceptance.
人脸身份为深度伪造检测提供了一个强大的信号。先前的研究表明,即使没有被明确建模,分类器也往往会隐式地学习到身份特征。这导致了相互冲突的观点:一些研究抑制身份线索以减少偏差,而另一些则依赖它们作为法医证据。为了调和这些观点,我们分析了两个假设:(1) 人脸身份本身是否足以用于检测深度伪造;(2) 这类身份特征在不同的操作方法下是否泛化能力较差。我们的实验确认身份信息是有用的但也是上下文相关的。虽然某些操纵保留了与身份一致的痕迹,但也有一些则扭曲了身份线索并损害了泛化性能。我们主张,不应盲目地抑制或依赖身份特征,而应该根据每个样本的相关性对其进行明确建模和自适应控制。 为此,我们提出了**SELFI**(Selective Fusion of Identity),这是一个可广泛应用于检测任务中的框架,能够动态调整对身份信息的使用方式。SELFI 包含两个主要部分:(1) 伪造感知的身份适配器 (FAIA),它从冻结的人脸识别模型中提取身份嵌入,并通过辅助监督将其投影到与伪造相关的空间;以及 (2) 身份感知融合模块 (IAFM),该模块使用基于相关性引导的机制选择性地结合身份和视觉特征。 在四个基准测试上的实验表明,SELFI 在跨操作泛化方面取得了提升,平均 AUC 比先前的方法提高了 3.1%。特别是在具有挑战性的 DFDC 数据集上,SELFI 的表现超过了之前的最佳结果 6%。待论文被接受后将公开代码。
https://arxiv.org/abs/2506.17592
Demographic bias in high-performance face recognition (FR) systems often eludes detection by existing metrics, especially with respect to subtle disparities in the tails of the score distribution. We introduce the Comprehensive Equity Index (CEI), a novel metric designed to address this limitation. CEI uniquely analyzes genuine and impostor score distributions separately, enabling a configurable focus on tail probabilities while also considering overall distribution shapes. Our extensive experiments (evaluating state-of-the-art FR systems, intentionally biased models, and diverse datasets) confirm CEI's superior ability to detect nuanced biases where previous methods fall short. Furthermore, we present CEI^A, an automated version of the metric that enhances objectivity and simplifies practical application. CEI provides a robust and sensitive tool for operational FR fairness assessment. The proposed methods have been developed particularly for bias evaluation in face biometrics but, in general, they are applicable for comparing statistical distributions in any problem where one is interested in analyzing the distribution tails.
高性能面部识别(FR)系统中的人口统计偏差往往被现有的度量标准所忽略,特别是在得分分布尾部的细微差异方面。我们引入了全面公平指数(Comprehensive Equity Index, CEI),这是一种旨在解决此限制的新颖度量标准。CEI独特地分别分析真实和冒名顶替者的分数分布,并允许对尾概率进行可配置的重点关注,同时考虑整体分布形态。我们的广泛实验(评估最先进的FR系统、故意带有偏见的模型以及多样化的数据集)证实了CEI在检测先前方法未能发现的细微偏差方面的卓越能力。此外,我们还提出了CEI^A,即该度量标准的自动化版本,它增强了客观性并简化了实际应用。CEI为操作层面的FR公平性评估提供了一个稳健且敏感的工具。所提出的方法特别为了面部生物识别中的偏见评估而开发,但一般来说,在任何关注分布尾部的统计分析问题中都适用。
https://arxiv.org/abs/2506.10564