Abstract
Biometric Verification (BV) systems often exhibit accuracy disparities across different demographic groups, leading to biases in BV applications. Assessing and quantifying these biases is essential for ensuring the fairness of BV systems. However, existing bias evaluation metrics in BV have limitations, such as focusing exclusively on match or non-match error rates, overlooking bias on demographic groups with performance levels falling between the best and worst performance levels, and neglecting the magnitude of the bias present. This paper presents an in-depth analysis of the limitations of current bias evaluation metrics in BV and, through experimental analysis, demonstrates their contextual suitability, merits, and limitations. Additionally, it introduces a novel general-purpose bias evaluation measure for BV, the ``Sum of Group Error Differences (SEDG)''. Our experimental results on controlled synthetic datasets demonstrate the effectiveness of demographic bias quantification when using existing metrics and our own proposed measure. We discuss the applicability of the bias evaluation metrics in a set of simulated demographic bias scenarios and provide scenario-based metric recommendations. Our code is publicly available under \url{this https URL}.
Abstract (translated)
生物特征验证（BV）系统通常在不同的 demographic群体之间表现出不一致的准确性，导致 BV 应用中的偏见。评估和量化这些偏见对于确保 BV 系统的公正性至关重要。然而，现有的 BV 偏见评估指标存在局限性，例如仅关注匹配或非匹配错误率，忽视了绩效水平在最好和最差水平之间的偏差，并忽略了存在的偏见的规模。本文对当前 BV 偏见评估指标的局限性进行了深入分析，并通过实验验证了它们的使用价值和局限性。此外，我们引入了一种新的通用的 BV 偏见评估指标——“组内误差差异（SEDG）”。我们对控制性合成数据集的实验结果表明，使用现有指标可以有效地量化 demographic 偏见。我们还讨论了在模拟 demographic 偏见场景中应用偏见评估指标的适用性，并提供了基于场景的指标建议。我们的代码可在 \url{这个链接} 上公开使用。
URL
https://arxiv.org/abs/2404.15385
