Face_Recognition
-
FaceLiVT: Face Recognition using Linear Vision Transformer with Structural Reparameterization For Mobile Device
2025-06-12 05:36:40Novendra Setyawan, Chi-Chia Sun, Mao-Hsiu Hsu, Wen-Kai Kuo, Jun-Wei HsiehAbstract
This paper introduces FaceLiVT, a lightweight yet powerful face recognition model that integrates a hybrid Convolution Neural Network (CNN)-Transformer architecture with an innovative and lightweight Multi-Head Linear Attention (MHLA) mechanism. By combining MHLA alongside a reparameterized token mixer, FaceLiVT effectively reduces computational complexity while preserving competitive accuracy. Extensive evaluations on challenging benchmarks; including LFW, CFP-FP, AgeDB-30, IJB-B, and IJB-C; highlight its superior performance compared to state-of-the-art lightweight models. MHLA notably improves inference speed, allowing FaceLiVT to deliver high accuracy with lower latency on mobile devices. Specifically, FaceLiVT is 8.6 faster than EdgeFace, a recent hybrid CNN-Transformer model optimized for edge devices, and 21.2 faster than a pure ViT-Based model. With its balanced design, FaceLiVT offers an efficient and practical solution for real-time face recognition on resource-constrained platforms.
Abstract (translated)
本文介绍了FaceLiVT，这是一种轻量级但强大的面部识别模型，它结合了混合卷积神经网络（CNN）-变压器架构，并且采用了创新的、轻量级的多头线性注意力（MHLA）机制。通过将MHLA与重新参数化的标记混合器结合使用，FaceLiVT有效地降低了计算复杂度，同时保持了竞争力的准确性。在包括LFW、CFP-FP、AgeDB-30、IJB-B和IJB-C在内的具有挑战性的基准测试中进行了广泛的评估，结果显示其性能优于最先进的轻量级模型。MHLA显著提高了推理速度，使FaceLiVT能够在移动设备上以较低的延迟提供高精度的结果。具体而言，FaceLiVT的速度比为边缘设备优化的最近混合CNN-Transformer模型EdgeFace快8.6倍，比基于纯ViT的模型快21.2倍。凭借其平衡的设计，FaceLiVT为资源受限平台上的实时面部识别提供了高效且实用的解决方案。
URL
https://arxiv.org/abs/2506.10361
PDF
-
ScoreMix: Improving Face Recognition via Score Composition in Diffusion Generators
2025-06-11 23:06:44Parsa Rahimi, Sebastien MarcelAbstract
In this paper, we propose ScoreMix, a novel yet simple data augmentation strategy leveraging the score compositional properties of diffusion models to enhance discriminator performance, particularly under scenarios with limited labeled data. By convexly mixing the scores from different class-conditioned trajectories during diffusion sampling, we generate challenging synthetic samples that significantly improve discriminative capabilities in all studied benchmarks. We systematically investigate class-selection strategies for mixing and discover that greater performance gains arise when combining classes distant in the discriminator's embedding space, rather than close in the generator's condition space. Moreover, we empirically show that, under standard metrics, the correlation between the generator's learned condition space and the discriminator's embedding space is minimal. Our approach achieves notable performance improvements without extensive parameter searches, demonstrating practical advantages for training discriminative models while effectively mitigating problems regarding collections of large datasets. Paper website: this https URL
Abstract (translated)
在这篇论文中，我们提出了ScoreMix，这是一种新颖且简单的数据增强策略，利用扩散模型的分数组合特性来提升判别器在标注数据有限场景下的性能。通过将不同类条件轨迹在扩散采样过程中得到的分数进行凸混合处理，我们生成了具有挑战性的合成样本，这些样本显著提升了所有研究基准中的判别能力。系统地调查了用于混合的类别选择策略，并发现当结合在判别器嵌入空间中距离较远而非生成器条件空间中接近的类时，可以获得更大的性能提升。此外，实证研究表明，在标准度量下，生成器学习到的条件空间与判别器的嵌入空间之间的相关性极小。我们的方法无需广泛的参数搜索即可实现显著的性能改进，展示了在训练判别模型方面的实用优势，并有效地缓解了关于大规模数据集收集的问题。 论文网站: [这里插入链接]
URL
https://arxiv.org/abs/2506.10226
PDF
-
Optimizing Genetic Algorithms with Multilayer Perceptron Networks for Enhancing TinyFace Recognition
2025-06-11 21:21:46Mohammad Subhi Al-Batah, Mowafaq Salem Alzboon, Muhyeeddin AlqaralehAbstract
This study conducts an empirical examination of MLP networks investigated through a rigorous methodical experimentation process involving three diverse datasets: TinyFace, Heart Disease, and Iris. Study Overview: The study includes three key methods: a) a baseline training using the default settings for the Multi-Layer Perceptron (MLP), b) feature selection using Genetic Algorithm (GA) based refinement c) Principal Component Analysis (PCA) based dimension reduction. The results show important information on how such techniques affect performance. While PCA had showed benefits in low-dimensional and noise-free datasets GA consistently increased accuracy in complex datasets by accurately identifying critical features. Comparison reveals that feature selection and dimensionality reduction play interdependent roles in enhancing MLP performance. The study contributes to the literature on feature engineering and neural network parameter optimization, offering practical guidelines for a wide range of machine learning tasks
Abstract (translated)
这项研究通过严格的系统实验过程对多层感知机（MLP）网络进行了实证考察，涉及三个不同的数据集：TinyFace、心脏病和鸢尾花。**研究概览**：该研究包括三种关键方法： a) 使用默认设置进行基线训练的Multi-Layer Perceptron (MLP) b) 采用遗传算法（GA）优化后的特征选择 c) 基于主成分分析（PCA）的降维 结果展示了这些技术如何影响性能的重要信息。在低维度和噪声较少的数据集中，PCA显示出其益处；而在复杂数据集上，基于GA的精确识别关键特征则持续提高了准确率。比较显示，特征选择与降维在增强MLP性能方面扮演着相互依赖的角色。 该研究为特征工程及神经网络参数优化领域的文献做出了贡献，并提供了适用于广泛机器学习任务的实际指导方针。
URL
https://arxiv.org/abs/2506.10184
PDF
-
Inverting Black-Box Face Recognition Systems via Zero-Order Optimization in Eigenface Space
2025-06-11 14:15:18Anton Razzhigaev, Matvey Mikhalchuk, Klim Kireev, Igor Udovichenko, Andrey Kuznetsov, Aleksandr PetiushkoAbstract
Reconstructing facial images from black-box recognition models poses a significant privacy threat. While many methods require access to embeddings, we address the more challenging scenario of model inversion using only similarity scores. This paper introduces DarkerBB, a novel approach that reconstructs color faces by performing zero-order optimization within a PCA-derived eigenface space. Despite this highly limited information, experiments on LFW, AgeDB-30, and CFP-FP benchmarks demonstrate that DarkerBB achieves state-of-the-art verification accuracies in the similarity-only setting, with competitive query efficiency.
Abstract (translated)
从黑盒识别模型中重构面部图像构成了重大的隐私威胁。尽管许多方法需要访问嵌入（embeddings），但我们解决了仅使用相似度分数进行模型反转的更具挑战性的情况。本文介绍了 DarkerBB，这是一种新颖的方法，它通过在由 PCA 导出的特征脸空间内执行零阶优化来重建彩色面部图像。尽管信息极为有限，在 LFW、AgeDB-30 和 CFP-FP 基准测试上的实验表明，DarkerBB 在仅基于相似度的情景下达到了最先进的验证准确率，并且具有竞争力的查询效率。
URL
https://arxiv.org/abs/2506.09777
PDF
-
Face recognition on point cloud with cgan-top for denoising
2025-06-07 17:09:31Junyu Liu, Jianfeng Ren, Sunhong Liang, Xudong JiangAbstract
Face recognition using 3D point clouds is gaining growing interest, while raw point clouds often contain a significant amount of noise due to imperfect sensors. In this paper, an end-to-end 3D face recognition on a noisy point cloud is proposed, which synergistically integrates the denoising and recognition modules. Specifically, a Conditional Generative Adversarial Network on Three Orthogonal Planes (cGAN-TOP) is designed to effectively remove the noise in the point cloud, and recover the underlying features for subsequent recognition. A Linked Dynamic Graph Convolutional Neural Network (LDGCNN) is then adapted to recognize faces from the processed point cloud, which hierarchically links both the local point features and neighboring features of multiple scales. The proposed method is validated on the Bosphorus dataset. It significantly improves the recognition accuracy under all noise settings, with a maximum gain of 14.81%.
Abstract (translated)
基于三维点云的面部识别技术正日益受到关注，然而原始点云数据通常因传感器不完善而包含大量噪声。本文提出了一种针对含噪点云的端到端3D面部识别方法，该方法协同集成了去噪和识别模块。 具体而言，设计了一个基于三个正交平面的条件生成对抗网络（cGAN-TOP），以有效去除点云中的噪声，并恢复后续识别所需的底层特征。接着采用了一种链接动态图卷积神经网络（LDGCNN）来从处理后的点云中识别人脸，该方法层次化地连接了局部点特征及其多尺度邻域特征。 所提出的方法在博斯普鲁斯数据集上进行了验证，在所有噪声设置下均显著提高了识别准确率，最高提升了14.81%。
URL
https://arxiv.org/abs/2506.06864
PDF
-
A Deep Learning Approach for Facial Attribute Manipulation and Reconstruction in Surveillance and Reconnaissance
2025-06-06 23:09:17Anees Nashath Shaik, Barbara Villarini, Vasileios ArgyriouAbstract
Surveillance systems play a critical role in security and reconnaissance, but their performance is often compromised by low-quality images and videos, leading to reduced accuracy in face recognition. Additionally, existing AI-based facial analysis models suffer from biases related to skin tone variations and partially occluded faces, further limiting their effectiveness in diverse real-world scenarios. These challenges are the results of data limitations and imbalances, where available training datasets lack sufficient diversity, resulting in unfair and unreliable facial recognition performance. To address these issues, we propose a data-driven platform that enhances surveillance capabilities by generating synthetic training data tailored to compensate for dataset biases. Our approach leverages deep learning-based facial attribute manipulation and reconstruction using autoencoders and Generative Adversarial Networks (GANs) to create diverse and high-quality facial datasets. Additionally, our system integrates an image enhancement module, improving the clarity of low-resolution or occluded faces in surveillance footage. We evaluate our approach using the CelebA dataset, demonstrating that the proposed platform enhances both training data diversity and model fairness. This work contributes to reducing bias in AI-based facial analysis and improving surveillance accuracy in challenging environments, leading to fairer and more reliable security applications.
Abstract (translated)
监控系统在安全和侦察中扮演着关键角色，但其性能常常因图像和视频质量低劣而受损，导致面部识别的准确性降低。此外，现有的基于人工智能的面部分析模型受肤色差异及部分遮挡脸部的影响而产生偏差，进一步限制了它们在多样化现实场景中的有效性。这些挑战源于数据限制和不平衡问题，现有训练数据集缺乏足够的多样性，从而导致不公平且不可靠的面部识别性能。 为了解决这些问题，我们提出了一种基于数据驱动平台的方法，通过生成合成训练数据来增强监控能力，并弥补数据集偏差。我们的方法利用深度学习技术进行面部属性操作与重建，使用自动编码器和生成对抗网络（GAN）创建多样化的高质量面部图像数据集。此外，该系统还整合了一个图像增强模块，以提高低分辨率或部分遮挡的面部在监控录像中的清晰度。 我们通过CelebA数据集验证了这种方法的有效性，结果表明所提出的平台可以提升训练数据多样性并改善模型公平性。这项工作有助于减少基于人工智能的面部分析中的偏差，并在充满挑战的环境中提升监控系统的准确性，从而为更公正和可靠的安防应用提供支持。
URL
https://arxiv.org/abs/2506.06578
PDF
-
Towards Large-Scale Pose-Invariant Face Recognition Using Face Defrontalization
2025-06-04 22:28:06Patrik Mesec, Alan Jovi\'cAbstract
Face recognition under extreme head poses is a challenging task. Ideally, a face recognition system should perform well across different head poses, which is known as pose-invariant face recognition. To achieve pose invariance, current approaches rely on sophisticated methods, such as face frontalization and various facial feature extraction model architectures. However, these methods are somewhat impractical in real-life settings and are typically evaluated on small scientific datasets, such as Multi-PIE. In this work, we propose the inverse method of face frontalization, called face defrontalization, to augment the training dataset of facial feature extraction model. The method does not introduce any time overhead during the inference step. The method is composed of: 1) training an adapted face defrontalization FFWM model on a frontal-profile pairs dataset, which has been preprocessed using our proposed face alignment method; 2) training a ResNet-50 facial feature extraction model based on ArcFace loss on a raw and randomly defrontalized large-scale dataset, where defrontalization was performed with our previously trained face defrontalization model. Our method was compared with the existing approaches on four open-access datasets: LFW, AgeDB, CFP, and Multi-PIE. Defrontalization shows improved results compared to models without defrontalization, while the proposed adjustments show clear superiority over the state-of-the-art face frontalization FFWM method on three larger open-access datasets, but not on the small Multi-PIE dataset for extreme poses (75 and 90 degrees). The results suggest that at least some of the current methods may be overfitted to small datasets.
Abstract (translated)
极端姿态下的面部识别是一个具有挑战性的任务。理想情况下，一个面部识别系统应该能够在不同的头部姿势下表现良好，这被称为无姿态变化的面部识别。为了实现这一目标，目前的方法依赖于复杂的技术，如面部正面化和各种面部特征提取模型架构。然而，在实际应用中这些方法往往不够实用，并且通常在小规模科学数据集（如Multi-PIE）上进行评估。 我们提出了一种逆向技术——面部非正面化(defrontalization)，用来增强面部特征提取模型的训练数据集。该方法不会增加推理步骤的时间开销。具体来说，我们的方法包括以下两个部分： 1. 在预先处理过的脸部对齐的数据集中（使用我们提出的面部对齐方法），通过训练一个自适应面部非正面化(FFWM)模型。 2. 基于ArcFace损失函数，在原始和随机非正面化的大型数据集上训练ResNet-50面部特征提取模型，其中非正面化过程由我们之前训练的面部非正面化模型执行。 我们将该方法与现有技术进行了比较，并在四个开源数据集中进行了测试：LFW、AgeDB、CFP以及Multi-PIE。实验结果显示，在三个更大的开放访问数据集上，我们的方法相对于没有进行非正面化的模型和最新的面部正面化FFWM方法表现出明显的优越性；但在极端姿势（75度和90度）下的小型Multi-PIE数据集中并没有显示出优势。 这些结果表明，至少部分当前的方法可能在小规模数据集上存在过拟合的问题。
URL
https://arxiv.org/abs/2506.04496
PDF
-
Beyond black and white: A more nuanced approach to facial recognition with continuous ethnicity labels
2025-06-02 10:49:36Pedro C. Neto, Naser Damer, Jaime S. Cardoso, Ana F. SequeiraAbstract
Bias has been a constant in face recognition models. Over the years, researchers have looked at it from both the model and the data point of view. However, their approach to mitigation of data bias was limited and lacked insight on the real nature of the problem. Here, in this document, we propose to revise our use of ethnicity labels as a continuous variable instead of a discrete value per identity. We validate our formulation both experimentally and theoretically, showcasing that not all identities from one ethnicity contribute equally to the balance of the dataset; thus, having the same number of identities per ethnicity does not represent a balanced dataset. We further show that models trained on datasets balanced in the continuous space consistently outperform models trained on data balanced in the discrete space. We trained more than 65 different models, and created more than 20 subsets of the original datasets.
Abstract (translated)
偏见一直是面部识别模型中的一个持续问题。多年来，研究人员从模型和数据两个角度对此进行了研究。然而，他们缓解数据偏差的方法是有限的，并且缺乏对问题本质的理解。在此文档中，我们提出将种族标签作为连续变量而非每个身份的一个离散值来使用。我们通过实验和理论验证了我们的方法，展示了来自同一族裔的不同身份对于数据集平衡的贡献并不相同；因此，拥有相同比例的各族裔身份并不代表一个均衡的数据集。此外，我们还证明在连续空间中平衡的数据集上训练出的模型始终优于在离散空间中平衡的数据集上训练出的模型。为了支持这一结论，我们训练了超过65种不同的模型，并创建了原始数据集的20多个子集。
URL
https://arxiv.org/abs/2506.01532
PDF
-
Efficient Estimation of Regularized Tyler's M-Estimator Using Approximate LOOCV
2025-05-30 16:43:14Karim Abou-MoustafaAbstract
We consider the problem of estimating a regularization parameter, or a shrinkage coefficient $\alpha \in (0,1)$ for Regularized Tyler's M-estimator (RTME). In particular, we propose to estimate an optimal shrinkage coefficient by setting $\alpha$ as the solution to a suitably chosen objective function; namely the leave-one-out cross-validated (LOOCV) log-likelihood loss. Since LOOCV is computationally prohibitive even for moderate sample size $n$, we propose a computationally efficient approximation for the LOOCV log-likelihood loss that eliminates the need for invoking the RTME procedure $n$ times for each sample left out during the LOOCV procedure. This approximation yields an $O(n)$ reduction in the running time complexity for the LOOCV procedure, which results in a significant speedup for computing the LOOCV estimate. We demonstrate the efficiency and accuracy of the proposed approach on synthetic high-dimensional data sampled from heavy-tailed elliptical distributions, as well as on real high-dimensional datasets for object recognition, face recognition, and handwritten digit's recognition. Our experiments show that the proposed approach is efficient and consistently more accurate than other methods in the literature for shrinkage coefficient estimation.
Abstract (translated)
我们考虑了估计正则化参数或Regularized Tyler的M-估计量(RTME)中的收缩系数$\alpha \in (0,1)$的问题。具体而言，我们建议通过将$\alpha$设置为一个合适的目标函数的解来估计最优的收缩系数；即，leave-one-out交叉验证(LOOCV)对数似然损失函数。由于即使对于中等样本量$n$，LOOCV在计算上也是不可行的，我们提出了一种用于LOOCV对数似然损失的计算效率高的近似方法，该方法消除了在LOOCV过程中每次留出一个样本时调用RTME程序$n$次的需求。这种近似方法将LOOCV过程的时间复杂度降低了$O(n)$，从而大大加快了LOOCV估计的计算速度。我们在从重型长尾椭圆分布中采样的合成高维数据以及用于物体识别、人脸识别和手写数字识别的实际高维数据集上展示了所提出的方法的有效性和准确性。我们的实验表明，与文献中的其他收缩系数估算方法相比，所提出的方法不仅效率更高，而且始终更准确。
URL
https://arxiv.org/abs/2505.24781
PDF
-
Leveraging Intermediate Features of Vision Transformer for Face Anti-Spoofing
2025-05-30 09:33:01Mika Feng, Koichi Ito, Takafumi Aoki, Tetsushi Ohki, Masakatsu NishigakiAbstract
Face recognition systems are designed to be robust against changes in head pose, illumination, and blurring during image capture. If a malicious person presents a face photo of the registered user, they may bypass the authentication process illegally. Such spoofing attacks need to be detected before face recognition. In this paper, we propose a spoofing attack detection method based on Vision Transformer (ViT) to detect minute differences between live and spoofed face images. The proposed method utilizes the intermediate features of ViT, which have a good balance between local and global features that are important for spoofing attack detection, for calculating loss in training and score in inference. The proposed method also introduces two data augmentation methods: face anti-spoofing data augmentation and patch-wise data augmentation, to improve the accuracy of spoofing attack detection. We demonstrate the effectiveness of the proposed method through experiments using the OULU-NPU and SiW datasets.
Abstract (translated)
面部识别系统旨在适应头部姿态、光照条件以及图像采集时的模糊变化。如果恶意人员使用已注册用户的面部照片，他们可能会非法绕过身份验证过程。此类冒充攻击需要在进行面部识别之前被检测出来。本文提出了一种基于视觉变换器（Vision Transformer, ViT）的冒充攻击检测方法，旨在通过细微差异区分真实人脸图像和伪造的人脸图像。 所提出的方法利用了ViT中间层特征，这些特征在局部和全局特征之间取得了良好的平衡，并且对识别冒充攻击非常重要。此方法将这些特征用于训练中的损失计算以及推理过程中的分数计算。此外，该方法还引入了两种数据增强技术：面部防伪造数据增强及块级数据增强，以提高冒充攻击检测的准确性。 通过使用OULU-NPU和SiW数据集进行实验，我们展示了所提出方法的有效性。
URL
https://arxiv.org/abs/2505.24402
PDF
-
50 Years of Automated Face Recognition
2025-05-30 06:10:48Minchul Kim, Anil Jain, Xiaoming LiuAbstract
Over the past 50 years, automated face recognition has evolved from rudimentary, handcrafted systems into sophisticated deep learning models that rival and often surpass human performance. This paper chronicles the history and technological progression of FR, from early geometric and statistical methods to modern deep neural architectures leveraging massive real and AI-generated datasets. We examine key innovations that have shaped the field, including developments in dataset, loss function, neural network design and feature fusion. We also analyze how the scale and diversity of training data influence model generalization, drawing connections between dataset growth and benchmark improvements. Recent advances have achieved remarkable milestones: state-of-the-art face verification systems now report False Negative Identification Rates of 0.13% against a 12.4 million gallery in NIST FRVT evaluations for 1:N visa-to-border matching. While recent advances have enabled remarkable accuracy in high- and low-quality face scenarios, numerous challenges persist. While remarkable progress has been achieved, several open research problems remain. We outline critical challenges and promising directions for future face recognition research, including scalability, multi-modal fusion, synthetic identity generation, and explainable systems.
Abstract (translated)
在过去50年里，自动面部识别技术已经从原始的手工制作系统发展成为复杂的深度学习模型，并且在许多情况下其性能已经超过人类。本文回顾了面部识别（FR）的历史和技术进步，从早期的几何和统计方法到现代利用大量真实数据和人工智能生成数据集的深层神经网络架构的发展历程。我们分析了塑造这一领域的重要创新，包括数据集、损失函数、神经网络设计以及特征融合等方面的发展。 文中还探讨了训练数据规模和多样性如何影响模型泛化能力，并且讨论了随着数据集的增长，基准测试改进之间的联系。最近的进步已经取得了令人瞩目的成果：在NIST FRVT评估中，针对1:N的签证到边境匹配场景下，最先进的面部验证系统报告的假阴性识别率（FNIR）仅为0.13%，而其对比的数据集包含1240万张图像。 尽管近期进展已经在高质量和低质量人脸场景中实现了显著的准确性提升，但仍存在许多挑战。虽然已经取得了令人瞩目的进步，但仍然有许多开放的研究问题尚未解决。我们概述了未来面部识别研究中的关键挑战及有前景的方向，包括可扩展性、多模式融合、合成身份生成以及可解释系统等方面。
URL
https://arxiv.org/abs/2505.24247
PDF
-
A Responsible Face Recognition Approach for Small and Mid-Scale Systems Through Personalized Neural Networks
2025-05-26 12:45:01Sebastian Gro{\ss}, Stefan Heindorf, Philipp Terh\"orstAbstract
Traditional face recognition systems rely on extracting fixed face representations, known as templates, to store and verify identities. These representations are typically generated by neural networks that often lack explainability and raise concerns regarding fairness and privacy. In this work, we propose a novel model-template (MOTE) approach that replaces vector-based face templates with small personalized neural networks. This design enables more responsible face recognition for small and medium-scale systems. During enrollment, MOTE creates a dedicated binary classifier for each identity, trained to determine whether an input face matches the enrolled identity. Each classifier is trained using only a single reference sample, along with synthetically balanced samples to allow adjusting fairness at the level of a single individual during enrollment. Extensive experiments across multiple datasets and recognition systems demonstrate substantial improvements in fairness and particularly in privacy. Although the method increases inference time and storage requirements, it presents a strong solution for small- and mid-scale applications where fairness and privacy are critical.
Abstract (translated)
传统的面部识别系统依赖于提取固定的面部表示，即模板，来存储和验证身份。这些表示通常由神经网络生成，但往往缺乏可解释性，并且在公平性和隐私方面存在担忧。在这项工作中，我们提出了一种新颖的模型-模板（MOTE）方法，该方法用小型个性化神经网络替代基于向量的面部模板。这种设计使小规模和中等规模系统中的面部识别更加负责任。 在注册阶段，MOTE为每个身份创建一个专用的二元分类器，并训练它来确定输入的面部是否与已注册的身份匹配。每个分类器仅使用单个参考样本进行训练，并结合合成平衡样本以允许在个人层面上调整公平性。广泛的跨多个数据集和识别系统的实验表明，在公平性和隐私方面有了显著改进。 尽管该方法增加了推理时间和存储需求，但对于小型和中型应用（这些应用对公平性和隐私有严格要求）来说，它提供了一个强有力的解决方案。
URL
https://arxiv.org/abs/2505.19920
PDF
-
ExpertGen: Training-Free Expert Guidance for Controllable Text-to-Face Generation
2025-05-22 20:09:21Liang Shi, Yun FuAbstract
Recent advances in diffusion models have significantly improved text-to-face generation, but achieving fine-grained control over facial features remains a challenge. Existing methods often require training additional modules to handle specific controls such as identity, attributes, or age, making them inflexible and resource-intensive. We propose ExpertGen, a training-free framework that leverages pre-trained expert models such as face recognition, facial attribute recognition, and age estimation networks to guide generation with fine control. Our approach uses a latent consistency model to ensure realistic and in-distribution predictions at each diffusion step, enabling accurate guidance signals to effectively steer the diffusion process. We show qualitatively and quantitatively that expert models can guide the generation process with high precision, and multiple experts can collaborate to enable simultaneous control over diverse facial aspects. By allowing direct integration of off-the-shelf expert models, our method transforms any such model into a plug-and-play component for controllable face generation.
Abstract (translated)
最近在扩散模型方面的进展显著提升了文本到人脸生成的效果，但实现对面部特征的精细控制仍是一个挑战。现有方法通常需要训练额外模块来处理特定控制因素，如身份、属性或年龄等，这使得它们变得不够灵活且资源消耗大。我们提出了一种名为ExpertGen的新框架，这是一个无需重新训练的方法，它利用预训练的专业模型（例如面部识别、面部属性识别和年龄估算网络）来指导生成过程，并实现精细的控制。 我们的方法使用一个潜在一致性模型，在每次扩散步骤中确保预测的真实性和分布合理性，从而使准确的引导信号能够有效地调节扩散过程。我们从定性和定量两方面展示了专家模型可以以高精度指导生成过程，并且多个专家模型可以协作同时对多方面的面部特征进行控制。通过允许直接集成现成的专业模型，我们的方法将任何这样的模型转化为可插拔组件，用于可控的人脸生成。 这种方法不仅提高了生成人脸图像的灵活性和效率，还为研究者提供了更方便、高效的方式来探索更多关于面部识别与合成的研究方向。
URL
https://arxiv.org/abs/2505.17256
PDF
-
Pose-invariant face recognition via feature-space pose frontalization
2025-05-22 09:01:01Nikolay Stanishev, Yuhang Lu, Touradj EbrahimiAbstract
Pose-invariant face recognition has become a challenging problem for modern AI-based face recognition systems. It aims at matching a profile face captured in the wild with a frontal face registered in a database. Existing methods perform face frontalization via either generative models or learning a pose robust feature representation. In this paper, a new method is presented to perform face frontalization and recognition within the feature space. First, a novel feature space pose frontalization module (FSPFM) is proposed to transform profile images with arbitrary angles into frontal counterparts. Second, a new training paradigm is proposed to maximize the potential of FSPFM and boost its performance. The latter consists of a pre-training and an attention-guided fine-tuning stage. Moreover, extensive experiments have been conducted on five popular face recognition benchmarks. Results show that not only our method outperforms the state-of-the-art in the pose-invariant face recognition task but also maintains superior performance in other standard scenarios.
Abstract (translated)
无姿态不变的人脸识别已成为现代基于AI的人脸识别系统面临的挑战性问题。该任务旨在将野外采集的侧脸与数据库中注册的正脸进行匹配。现有的方法通过生成模型或学习鲁棒的姿态特征表示来进行人脸正面化。本文提出了一种新的方法，在特征空间内执行人脸正面化和识别。首先，提出了一个新颖的特征空间姿态正面化模块（FSPFM），将任意角度的侧脸图像转换为正脸对应物。其次，提出了一种新的训练范式来最大化FSPFM的潜力并提升其性能。后者包括预训练和注意力引导微调两个阶段。此外，在五个流行的人脸识别基准数据集上进行了广泛的实验。结果显示，不仅我们提出的方法在无姿态不变人脸识别人脸任务中优于现有最佳方法，并且在其他标准场景中也保持了优越的性能。
URL
https://arxiv.org/abs/2505.16412
PDF
-
Pura: An Efficient Privacy-Preserving Solution for Face Recognition
2025-05-21 12:50:25Guotao Xu, Bowen Zhao, Yang Xiao, Yantao Zhong, Liang Zhai, Qingqi PeiAbstract
Face recognition is an effective technology for identifying a target person by facial images. However, sensitive facial images raises privacy concerns. Although privacy-preserving face recognition is one of potential solutions, this solution neither fully addresses the privacy concerns nor is efficient enough. To this end, we propose an efficient privacy-preserving solution for face recognition, named Pura, which sufficiently protects facial privacy and supports face recognition over encrypted data efficiently. Specifically, we propose a privacy-preserving and non-interactive architecture for face recognition through the threshold Paillier cryptosystem. Additionally, we carefully design a suite of underlying secure computing protocols to enable efficient operations of face recognition over encrypted data directly. Furthermore, we introduce a parallel computing mechanism to enhance the performance of the proposed secure computing protocols. Privacy analysis demonstrates that Pura fully safeguards personal facial privacy. Experimental evaluations demonstrate that Pura achieves recognition speeds up to 16 times faster than the state-of-the-art.
Abstract (translated)
面部识别技术通过面部图像来识别人脸，是一种有效的方法。然而，敏感的面部图像是隐私保护的一大隐患。虽然隐私保护型面部识别技术是潜在解决方案之一，但它们既未能充分解决隐私问题，又不够高效。为此，我们提出了一种高效的隐私保护方案Pura，该方案能够有效地保护面部隐私，并支持在加密数据上进行有效的面部识别操作。 具体来说，通过门限Paillier密码系统，我们设计了一个私密性保护且无需交互的面部识别架构。此外，为了直接对加密后的数据执行面部识别操作，我们精心设计了一套底层的安全计算协议。并且，我们还引入了并行计算机制以增强所提出的安全计算协议的性能。 隐私分析显示Pura能够全面保障个人面部信息的安全性。实验评估表明，与现有的前沿技术相比，Pura在识别速度方面最多可快达16倍。
URL
https://arxiv.org/abs/2505.15476
PDF
-
DiffProb: Data Pruning for Face Recognition
2025-05-21 08:49:07Eduarda Caldeira, Jan Niklas Kolf, Naser Damer, Fadi BoutrosAbstract
Face recognition models have made substantial progress due to advances in deep learning and the availability of large-scale datasets. However, reliance on massive annotated datasets introduces challenges related to training computational cost and data storage, as well as potential privacy concerns regarding managing large face datasets. This paper presents DiffProb, the first data pruning approach for the application of face recognition. DiffProb assesses the prediction probabilities of training samples within each identity and prunes the ones with identical or close prediction probability values, as they are likely reinforcing the same decision boundaries, and thus contribute minimally with new information. We further enhance this process with an auxiliary cleaning mechanism to eliminate mislabeled and label-flipped samples, boosting data quality with minimal loss. Extensive experiments on CASIA-WebFace with different pruning ratios and multiple benchmarks, including LFW, CFP-FP, and IJB-C, demonstrate that DiffProb can prune up to 50% of the dataset while maintaining or even, in some settings, improving the verification accuracies. Additionally, we demonstrate DiffProb's robustness across different architectures and loss functions. Our method significantly reduces training cost and data volume, enabling efficient face recognition training and reducing the reliance on massive datasets and their demanding management.
Abstract (translated)
面部识别模型由于深度学习的进步和大规模数据集的可用性而取得了显著进展。然而，依赖于大量标注数据集引入了训练计算成本高、存储需求大以及处理大规模人脸数据集时可能出现的数据隐私问题等挑战。本文介绍了DiffProb，这是第一个面向人脸识别应用的数据精简方法。DiffProb评估每个身份内训练样本的预测概率，并修剪掉那些具有相同或接近预测概率值的样本，因为这些样本很可能是重复强化相同的决策边界，因此贡献的新信息量很小。此外，我们还通过辅助清理机制来消除错误标注和标签翻转的样本，从而在几乎不损失数据的情况下提高数据质量。 我们在CASIA-WebFace上进行了广泛的实验，涵盖了不同的精简比率以及包括LFW、CFP-FP和IJB-C在内的多个基准测试。结果显示，DiffProb可以在保持或甚至在某些情况下提升验证准确率的同时最多修剪掉50%的数据集。此外，我们还展示了DiffProb在不同架构和损失函数下具有良好的鲁棒性。 我们的方法显著降低了训练成本和数据量，使高效的人脸识别训练成为可能，并减少了对大规模数据集及其复杂管理的依赖。
URL
https://arxiv.org/abs/2505.15272
PDF
-
Facial Recognition Leveraging Generative Adversarial Networks
2025-05-17 07:26:12Zhongwen Li, Zongwei Li, Xiaoqi LiAbstract
Face recognition performance based on deep learning heavily relies on large-scale training data, which is often difficult to acquire in practical applications. To address this challenge, this paper proposes a GAN-based data augmentation method with three key contributions: (1) a residual-embedded generator to alleviate gradient vanishing/exploding problems, (2) an Inception ResNet-V1 based FaceNet discriminator for improved adversarial training, and (3) an end-to-end framework that jointly optimizes data generation and recognition performance. Experimental results demonstrate that our approach achieves stable training dynamics and significantly improves face recognition accuracy by 12.7% on the LFW benchmark compared to baseline methods, while maintaining good generalization capability with limited training samples.
Abstract (translated)
基于深度学习的面部识别性能很大程度上依赖于大规模训练数据，而这些数据在实际应用中往往难以获得。为了解决这一挑战，本文提出了一种基于生成对抗网络（GAN）的数据增强方法，该方法有三个关键贡献：(1) 嵌入残差的生成器以缓解梯度消失和爆炸问题；(2) 一个基于Inception ResNet-V1的FaceNet判别器，用于改进对抗训练；(3) 一种端到端框架，它同时优化数据生成和识别性能。实验结果表明，我们的方法实现了稳定的训练动力学，并且在LFW基准测试上相比基线方法将面部识别准确率提高了12.7%，同时还保持了在有限样本下的良好泛化能力。
URL
https://arxiv.org/abs/2505.11884
PDF
-
MorphGuard: Morph Specific Margin Loss for Enhancing Robustness to Face Morphing Attacks
2025-05-15 17:00:16Iurii Medvedev, Nuno GoncalvesAbstract
Face recognition has evolved significantly with the advancement of deep learning techniques, enabling its widespread adoption in various applications requiring secure authentication. However, this progress has also increased its exposure to presentation attacks, including face morphing, which poses a serious security threat by allowing one identity to impersonate another. Therefore, modern face recognition systems must be robust against such attacks. In this work, we propose a novel approach for training deep networks for face recognition with enhanced robustness to face morphing attacks. Our method modifies the classification task by introducing a dual-branch classification strategy that effectively handles the ambiguity in the labeling of face morphs. This adaptation allows the model to incorporate morph images into the training process, improving its ability to distinguish them from bona fide samples. Our strategy has been validated on public benchmarks, demonstrating its effectiveness in enhancing robustness against face morphing attacks. Furthermore, our approach is universally applicable and can be integrated into existing face recognition training pipelines to improve classification-based recognition methods.
Abstract (translated)
面部识别技术随着深度学习技术的进步而显著发展，使其在需要安全认证的各种应用中得以广泛应用。然而，这种进步也增加了它对展示攻击（包括人脸融合）的暴露风险，这种攻击允许一个身份冒充另一个身份，从而构成了严重的安全威胁。因此，现代面部识别系统必须具备抵御此类攻击的能力。 在此研究中，我们提出了一种新颖的方法，用于训练深度网络进行面部识别，并增强其对抗面部融合攻击的鲁棒性。我们的方法通过引入双分支分类策略来修改分类任务，这种策略有效地处理了人脸融合标签中的模糊性。这一调整使模型能够将融合图像纳入训练过程，从而提高其区分这些图像与真实样本的能力。 我们的策略已经在公开基准测试中得到了验证，证明它在增强抵御面部融合攻击的鲁棒性方面是有效的。此外，我们的方法具有普遍适用性，并可以整合到现有的面部识别训练流程中，以改进基于分类的方法。
URL
https://arxiv.org/abs/2505.10497
PDF
-
StoryReasoning Dataset: Using Chain-of-Thought for Scene Understanding and Grounded Story Generation
2025-05-15 13:42:14Daniel A. P. Oliveira, David Martins de MatosAbstract
Visual storytelling systems struggle to maintain character identity across frames and link actions to appropriate subjects, frequently leading to referential hallucinations. These issues can be addressed through grounding of characters, objects, and other entities on the visual elements. We propose StoryReasoning, a dataset containing 4,178 stories derived from 52,016 movie images, with both structured scene analyses and grounded stories. Each story maintains character and object consistency across frames while explicitly modeling multi-frame relationships through structured tabular representations. Our approach features cross-frame object re-identification using visual similarity and face recognition, chain-of-thought reasoning for explicit narrative modeling, and a grounding scheme that links textual elements to visual entities across multiple frames. We establish baseline performance by fine-tuning Qwen2.5-VL 7B, creating Qwen Storyteller, which performs end-to-end object detection, re-identification, and landmark detection while maintaining consistent object references throughout the story. Evaluation demonstrates a reduction from 4.06 to 3.56 (-12.3%) hallucinations on average per story when compared to a non-fine-tuned model.
Abstract (translated)
视觉叙事系统在维护角色身份以及将动作与适当主体关联方面面临挑战，这常常导致指代幻觉。这些问题可以通过基于视觉元素对人物、物体和其他实体进行定位来解决。我们提出了StoryReasoning数据集，包含从52,016张电影图片中提取的4,178个故事，并提供了结构化的场景分析和基于视觉定位的故事文本。每个故事在不同帧之间保持了角色和对象的一致性，并通过结构化表格表示显式地建模多帧之间的关系。 我们的方法包括利用视觉相似度和面部识别进行跨帧对象重新识别，链式思维推理用于明确叙述模型构建，以及将文本元素链接到多个帧中的视觉实体的定位方案。我们通过对Qwen2.5-VL 7B进行微调来建立基线性能，创建了Qwen讲故事模型，该模型在故事中保持一致的对象引用的同时执行端到端的对象检测、重新识别和地标检测。评估结果表明，与未经微调的模型相比，平均每个故事中的幻觉减少了12.3%（从4.06减少到3.56）。
URL
https://arxiv.org/abs/2505.10292
PDF
-
Test-Time Augmentation for Pose-invariant Face Recognition
2025-05-14 10:11:35Jaemin Jung, Youngjoon Jang, Joon Son ChungAbstract
The goal of this paper is to enhance face recognition performance by augmenting head poses during the testing phase. Existing methods often rely on training on frontalised images or learning pose-invariant representations, yet both approaches typically require re-training and testing for each dataset, involving a substantial amount of effort. In contrast, this study proposes Pose-TTA, a novel approach that aligns faces at inference time without additional training. To achieve this, we employ a portrait animator that transfers the source image identity into the pose of a driving image. Instead of frontalising a side-profile face -- which can introduce distortion -- Pose-TTA generates matching side-profile images for comparison, thereby reducing identity information loss. Furthermore, we propose a weighted feature aggregation strategy to address any distortions or biases arising from the synthetic data, thus enhancing the reliability of the augmented images. Extensive experiments on diverse datasets and with various pre-trained face recognition models demonstrate that Pose-TTA consistently improves inference performance. Moreover, our method is straightforward to integrate into existing face recognition pipelines, as it requires no retraining or fine-tuning of the underlying recognition models.
Abstract (translated)
本文的目标是通过在测试阶段增强头部姿态来提高面部识别的性能。现有的方法通常依赖于使用正面化的图像进行训练或学习位置不变表示，然而这两种方法通常都需要为每个数据集重新进行训练和测试，涉及大量的工作量。相比之下，本研究提出了一种名为Pose-TTA的新方法，在推理时对齐面部而无需额外训练。 为了实现这一目标，我们采用了肖像动画师来将源图像的身份转移到驱动图像的姿态中。与正面化侧面轮廓脸不同（这可能会引入失真），Pose-TTA生成用于比较的匹配侧面轮廓图像，从而减少身份信息损失。 此外，我们提出了一种加权特征聚合策略，以解决由合成数据引起的任何失真或偏差问题，从而使增强图像更加可靠。 在各种数据集和不同的预训练面部识别模型上进行的大量实验表明，Pose-TTA可以一致地提高推理性能。而且，我们的方法易于集成到现有的面部识别管道中，因为它不需要对底层识别模型进行重新训练或微调。
URL
https://arxiv.org/abs/2505.09256
PDF
