Abstract
Generative Adversarial Network approaches such as StyleGAN/2 provide two key benefits: the ability to generate photo-realistic face images and possessing a semantically structured latent space from which these images are created. Many approaches have emerged for editing images derived from vectors in the latent space of a pre-trained StyleGAN/2 models by identifying semantically meaningful directions (e.g., gender or age) in the latent space. By moving the vector in a specific direction, the ideal result would only change the target feature while preserving all the other features. Providing an ideal data augmentation approach for gesture research as it could be used to generate numerous image variations whilst keeping the facial expressions intact. However, entanglement issues, where changing one feature inevitably affects other features, impacts the ability to preserve facial expressions. To address this, we propose the use of an addition to the loss function of a Facial Keypoint Detection model to restrict changes to the facial expressions. Building on top of an existing model, adding the proposed Human Face Landmark Detection (HFLD) loss, provided by a pre-trained Facial Keypoint Detection model, to the original loss function. We quantitatively and qualitatively evaluate the existing and our extended model, showing the effectiveness of our approach in addressing the entanglement issue and maintaining the facial expression. Our approach achieves up to 49% reduction in the change of emotion in our experiments. Moreover, we show the benefit of our approach by comparing with state-of-the-art models. By increasing the ability to preserve the facial gesture and expression during facial transformation, we present a way to create human face images with fixed expression but different appearances, making it a reliable data augmentation approach for Facial Gesture and Expression research.
Abstract (translated)
生成对抗网络（如StyleGAN/2）提供了两个关键优势：能够生成逼真的面部图像以及拥有语义结构化的潜在空间，从中可以创建这些图像。许多方法已经出现，通过识别预训练的StyleGAN/2模型中的潜在空间中具有特定语义意义的方向（例如性别或年龄），来编辑从该空间向量衍生出的图片。理想的结果是移动向量以仅改变目标特征而不影响其他所有特征。这种数据增强方法为手势研究提供了理想的途径，因为它可以生成许多图像变化的同时保持面部表情不变。然而，纠缠问题的存在使得在改变一个特性时不可避免地会影响到其它特性，从而难以保留面部表情。 为了应对这一挑战，我们建议通过向面部关键点检测模型的损失函数中添加一项来限制对表情的变化。基于现有的模型，在原始损失函数的基础上增加我们提出的“人类面部地标检测”（HFLD）损失项，该损失项来自预训练的关键点检测模型。通过对现有模型和扩展后的模型进行定量和定性评估，证明了我们的方法在解决纠缠问题并保持面部表情方面是有效的。实验结果表明，在我们的实验中，情感变化减少了多达49%。 此外，我们通过与最先进的模型进行比较展示了我们方法的优势。通过增强在面部转换过程中保留面部手势和表达的能力，我们可以创建具有固定表情但不同外观的人脸图像，从而为面部手势和表情研究提供可靠的增广数据方法。
URL
https://arxiv.org/abs/2505.06436
