Abstract
3D textured face reconstruction from sketches applicable in many scenarios such as animation, 3D avatars, artistic design, missing people search, etc., is a highly promising but underdeveloped research topic. On the one hand, the stylistic diversity of sketches leads to existing sketch-to-3D-face methods only being able to handle pose-limited and realistically shaded sketches. On the other hand, texture plays a vital role in representing facial appearance, yet sketches lack this information, necessitating additional texture control in the reconstruction process. This paper proposes a novel method for reconstructing controllable textured and detailed 3D faces from sketches, named S2TD-Face. S2TD-Face introduces a two-stage geometry reconstruction framework that directly reconstructs detailed geometry from the input sketch. To keep geometry consistent with the delicate strokes of the sketch, we propose a novel sketch-to-geometry loss that ensures the reconstruction accurately fits the input features like dimples and wrinkles. Our training strategies do not rely on hard-to-obtain 3D face scanning data or labor-intensive hand-drawn sketches. Furthermore, S2TD-Face introduces a texture control module utilizing text prompts to select the most suitable textures from a library and seamlessly integrate them into the geometry, resulting in a 3D detailed face with controllable texture. S2TD-Face surpasses existing state-of-the-art methods in extensive quantitative and qualitative experiments. Our project is available at this https URL .
Abstract (translated)
3D纹理面部重构是从许多场景中适用于动画、3D角色、艺术设计、失踪人员搜索等应用的草图。这是一个具有很高前景但尚未得到充分发展的研究课题。一方面，草图的多样化导致了只有能够处理姿态有限且现实主义着色的草图才能应用现有方法。另一方面，纹理在表示面部外观方面起着至关重要的作用，然而草图却没有这种信息，因此在重构过程中需要额外的纹理控制。本文提出了一个名为S2TD-Face的新方法，可以从草图中重构可控制纹理和详细信息的3D面部。S2TD-Face引入了一个两阶段几何重建框架，直接从输入草图重构详细的几何。为了保持几何与草图的精致线条一致，我们提出了一个新的从草图到几何的损失函数，确保重构准确地匹配输入特征，如皱纹和痘痘。我们的训练策略不依赖于难以获得的3D面部扫描数据或费力的人工手绘草图。此外，S2TD-Face还引入了一个纹理控制模块，利用文本提示从库中选择最合适的纹理，并将其无缝集成到几何中，从而实现具有可控制纹理的3D详细面部。S2TD-Face在广泛的定量实验和定性实验中超越了现有最先进的方法。我们的项目URL为：https://www. this URL.
URL
https://arxiv.org/abs/2408.01218
