Abstract
Face swapping combines one face's identity with another face's non-appearance attributes (expression, head pose, lighting) to generate a synthetic face. This technology is rapidly improving, but falls flat when reconstructing some attributes, particularly gaze. Image-based loss metrics that consider the full face do not effectively capture the perceptually important, yet spatially small, eye regions. Improving gaze in face swaps can improve naturalness and realism, benefiting applications in entertainment, human computer interaction, and more. Improved gaze will also directly improve Deepfake detection efforts, serving as ideal training data for classifiers that rely on gaze for classification. We propose a novel loss function that leverages gaze prediction to inform the face swap model during training and compare against existing methods. We find all methods to significantly benefit gaze in resulting face swaps.
Abstract (translated)
脸交换将一个人脸的身份与另一个人脸的非出现属性(表情、头部姿势、照明)生成一个合成人脸。这项技术正在迅速发展，但在某些属性的重建方面表现平平，特别是眼睛区域。考虑整个面部的损失度量并没有有效捕捉感知上重要但空间上较小的眼睛区域。改善脸交换中的眼睛区域可以改善自然性和真实感，受益于娱乐、人机交互和其他应用领域。改善眼睛区域将直接改善 Deepfake 检测努力，作为依赖于眼睛识别的分类器的理想训练数据。我们提议一种新损失函数，利用眼睛预测在训练期间通知脸交换模型，并与其他方法进行比较。我们发现所有方法都显著地受益于最终脸交换中的眼睛区域。
URL
https://arxiv.org/abs/2305.16138
