Abstract
Recent facial texture generation methods prefer to use deep networks to synthesize image content and then fill in the UV map, thus generating a compelling full texture from a single image. Nevertheless, the synthesized texture UV map usually comes from a space constructed by the training data or the 2D face generator, which limits the methods' generalization ability for in-the-wild input images. Consequently, their facial details, structures and identity may not be consistent with the input. In this paper, we address this issue by proposing a style transfer-based facial texture refinement method named FaceRefiner. FaceRefiner treats the 3D sampled texture as style and the output of a texture generation method as content. The photo-realistic style is then expected to be transferred from the style image to the content image. Different from current style transfer methods that only transfer high and middle level information to the result, our style transfer method integrates differentiable rendering to also transfer low level (or pixel level) information in the visible face regions. The main benefit of such multi-level information transfer is that, the details, structures and semantics in the input can thus be well preserved. The extensive experiments on Multi-PIE, CelebA and FFHQ datasets demonstrate that our refinement method can improve the texture quality and the face identity preserving ability, compared with state-of-the-arts.
Abstract (translated)
最近的面部纹理生成方法倾向于使用深度网络合成图像内容，然后填充UV映射（一种在三维建模中用于描述物体表面几何和材质分布的方法），从而从单张图片生成逼真的完整纹理。然而，所合成的纹理UV映射通常源自于训练数据或二维人脸生成器构建的空间内，这限制了这些方法对野外输入图像的泛化能力。因此，产生的面部细节、结构及身份可能与输入不一致。 为了解决这个问题，本文提出了一种基于风格迁移的面部纹理精炼方法，名为FaceRefiner。FaceRefiner将从3D采样的纹理视作风格，而纹理生成方法的输出则被视为内容。期望通过这种方法将照片般真实的风格从风格图像转移到内容图像上。与当前仅转移高层和中层信息到结果的风格迁移方法不同的是，我们的风格迁移方法结合了可微分渲染技术，从而也能够将低层（或像素级别）的信息传递至可见面部区域。这种多层次信息传输的主要优点在于，可以更好地保留输入中的细节、结构及语义。 在Multi-PIE、CelebA和FFHQ数据集上进行的广泛实验表明，与现有先进方法相比，我们的精炼方法能够提高纹理质量和人脸身份保持能力。
URL
https://arxiv.org/abs/2601.04520
