Abstract
Forensic sketch-to-mugshot matching is a challenging task in face recognition, primarily hindered by the scarcity of annotated forensic sketches and the modality gap between sketches and photographs. To address this, we propose CLIP4Sketch, a novel approach that leverages diffusion models to generate a large and diverse set of sketch images, which helps in enhancing the performance of face recognition systems in sketch-to-mugshot matching. Our method utilizes Denoising Diffusion Probabilistic Models (DDPMs) to generate sketches with explicit control over identity and style. We combine CLIP and Adaface embeddings of a reference mugshot, along with textual descriptions of style, as the conditions to the diffusion model. We demonstrate the efficacy of our approach by generating a comprehensive dataset of sketches corresponding to mugshots and training a face recognition model on our synthetic data. Our results show significant improvements in sketch-to-mugshot matching accuracy over training on an existing, limited amount of real face sketch data, validating the potential of diffusion models in enhancing the performance of face recognition systems across modalities. We also compare our dataset with datasets generated using GAN-based methods to show its superiority.
Abstract (translated)
刑事画像转储与 mug 照匹配是一项具有挑战性的任务，主要受到缺乏注释的刑事画像和画像之间的模态差距的阻碍。为解决这一问题，我们提出了 CLIP4Sketch，一种利用扩散模型生成大量且多样化的素描图像的新颖方法，这有助于提高在素描-到- mug 照匹配中脸部识别系统的性能。我们的方法利用狄诺伊散度概率分布模型（DDPMs）生成具有明确身份和风格控制的可扩展素描。我们将 CLIP 和 Adaface 的嵌入式参考 mug 照与风格描述文本作为扩散模型的条件。我们通过生成与 mug 照匹配的全面数据集来展示我们方法的有效性，并对扩散模型在跨模态增强面部识别系统性能方面的潜力进行了验证。我们的结果表明，与在有限真实面部素描数据上训练相比，我们的方法在素描-到- mug 照匹配准确性方面取得了显著的提高，验证了扩散模型的潜力在提高跨模态面部识别系统的性能方面。我们还比较了我们的数据集与使用基于 GAN 的方法生成的数据集，以显示其优越性。
URL
https://arxiv.org/abs/2408.01233
