Abstract
Despite rapid advances in computer graphics, creating high-quality photo-realistic virtual portraits is prohibitively expensive. Furthermore, the well-know ''uncanny valley'' effect in rendered portraits has a significant impact on the user experience, especially when the depiction closely resembles a human likeness, where any minor artifacts can evoke feelings of eeriness and repulsiveness. In this paper, we present a novel photo-realistic portrait generation framework that can effectively mitigate the ''uncanny valley'' effect and improve the overall authenticity of rendered portraits. Our key idea is to employ transfer learning to learn an identity-consistent mapping from the latent space of rendered portraits to that of real portraits. During the inference stage, the input portrait of an avatar can be directly transferred to a realistic portrait by changing its appearance style while maintaining the facial identity. To this end, we collect a new dataset, Daz-Rendered-Faces-HQ (DRFHQ), that is specifically designed for rendering-style portraits. We leverage this dataset to fine-tune the StyleGAN2 generator, using our carefully crafted framework, which helps to preserve the geometric and color features relevant to facial identity. We evaluate our framework using portraits with diverse gender, age, and race variations. Qualitative and quantitative evaluations and ablation studies show the advantages of our method compared to state-of-the-art approaches.
Abstract (translated)
尽管计算机图形学取得了快速进展，但创建高质量的照片现实主义的虚拟肖像可能过于昂贵。此外，已知的“奇异谷”效应在渲染肖像中对用户体验的影响很大，尤其是在描述非常类似于人类肖像的时候，任何细小瑕疵都可能引发奇异和令人厌恶的感觉。在本文中，我们提出了一个新颖的照片现实主义肖像生成框架，可以有效地减轻“奇异谷”效应，提高渲染肖像的总体真实感。我们的关键想法是使用迁移学习从渲染肖像的潜在空间中学到与真实肖像的相似身份一致的映射。在推理阶段，可以通过改变虚拟角色的外观风格来直接将其转移到真实角色上。为此，我们收集了一个专门为渲染风格肖像设计的全新数据集Daz-Rendered-Faces-HQ（DRFHQ）。我们利用这个数据集来微调StyleGAN2生成器，并使用我们精心设计的框架，该框架有助于保留与面部身份相关的几何和色彩特征。我们通过评估具有不同性别、年龄和种族的肖像来评估我们的方法。定性和定量评估以及消融研究结果表明，与最先进的解决方案相比，我们的方法具有优势。
URL
https://arxiv.org/abs/2310.04194
