Abstract
3D face reconstruction from a single sketch is a critical yet underexplored task with significant practical applications. The primary challenges stem from the substantial modality gap between 2D sketches and 3D facial structures, including: (1) accurately extracting facial keypoints from 2D sketches; (2) preserving diverse facial expressions and fine-grained texture details; and (3) training a high-performing model with limited data. In this paper, we propose Sketch-1-to-3, a novel framework for realistic 3D face reconstruction from a single sketch, to address these challenges. Specifically, we first introduce the Geometric Contour and Texture Detail (GCTD) module, which enhances the extraction of geometric contours and texture details from facial sketches. Additionally, we design a deep learning architecture with a domain adaptation module and a tailored loss function to align sketches with the 3D facial space, enabling high-fidelity expression and texture reconstruction. To facilitate evaluation and further research, we construct SketchFaces, a real hand-drawn facial sketch dataset, and Syn-SketchFaces, a synthetic facial sketch dataset. Extensive experiments demonstrate that Sketch-1-to-3 achieves state-of-the-art performance in sketch-based 3D face reconstruction.
Abstract (translated)
从单幅素描进行三维人脸重建是一项具有重要实际应用价值但研究较少的任务。该任务的主要挑战在于二维素描与三维面部结构之间的显著模态差异，具体包括：（1）准确提取二维素描中的面部关键点；（2）保留多样化的面部表情和精细的纹理细节；以及（3）在数据有限的情况下训练高性能模型。为此，在本文中我们提出了Sketch-1-to-3框架，这是一种全新的从单幅素描生成逼真的三维人脸重建的方法，旨在解决上述挑战。 具体而言，我们首先引入了几何轮廓与纹理细节(GCTD)模块，该模块能够增强从面部素描中提取几何轮廓和纹理细节的能力。此外，为了使素描与其对应的3D面部空间对齐，我们设计了一个深度学习架构，其中包含领域适应模块及定制的损失函数，从而实现高保真度的表情与纹理重建。 为促进评估以及进一步的研究工作，我们构建了SketchFaces数据集，该数据集中包含了真实的素描人脸，并且还建立了一个合成的人脸素描数据集Syn-SketchFaces。通过广泛的实验表明，Sketch-1-to-3在基于素描的三维人脸识别任务上达到了最先进的性能水平。
URL
https://arxiv.org/abs/2502.17852
