Abstract
In this paper, we address the challenge of making ViT models more robust to unseen affine transformations. Such robustness becomes useful in various recognition tasks such as face recognition when image alignment failures occur. We propose a novel method called KP-RPE, which leverages key points (e.g.~facial landmarks) to make ViT more resilient to scale, translation, and pose variations. We begin with the observation that Relative Position Encoding (RPE) is a good way to bring affine transform generalization to ViTs. RPE, however, can only inject the model with prior knowledge that nearby pixels are more important than far pixels. Keypoint RPE (KP-RPE) is an extension of this principle, where the significance of pixels is not solely dictated by their proximity but also by their relative positions to specific keypoints within the image. By anchoring the significance of pixels around keypoints, the model can more effectively retain spatial relationships, even when those relationships are disrupted by affine transformations. We show the merit of KP-RPE in face and gait recognition. The experimental results demonstrate the effectiveness in improving face recognition performance from low-quality images, particularly where alignment is prone to failure. Code and pre-trained models are available.
Abstract (translated)
在本文中，我们解决了使ViT模型对未见到的平移变换更加鲁棒的问题。这种鲁棒性在各种识别任务中变得有用，例如在面部识别中，当图像对齐失败时。我们提出了一种名为KP-RPE的新方法，它利用关键点（例如~面部关键点）使ViT更加弹性，对平移和姿态变化具有鲁棒性。我们首先观察到，相对位置编码（RPE）是将平移变换推广到ViT的好的方法。然而，RPE只能将模型注入到附近像素比远距离像素更重要这一先验知识。关键点RPE（KP-RPE）是这一原则的扩展，其中像素的重要性不仅由其邻近性决定，还由其与图像中特定关键点之间的相对位置决定。通过将像素的重要性锚定在关键点上，模型可以在平移变换破坏时更有效地保留空间关系。我们在面部和步态识别中展示了KP-RPE的优点。实验结果表明，从低质量图像中提高面部识别性能的有效方法，特别是在对齐容易失败的地方。代码和预训练模型可获得。
URL
https://arxiv.org/abs/2403.14852
