Abstract
Generating high-quality, photorealistic textures for 3D human avatars remains a fundamental yet challenging task in computer vision and multimedia field. However, real paired front and back images of human subjects are rarely available with privacy, ethical and cost of acquisition, which restricts scalability of the data. Additionally, learning priors from image inputs using deep generative models, such as GANs or diffusion models, to infer unseen regions such as the human back often leads to artifacts, structural inconsistencies, or loss of fine-grained detail. To address these issues, we present SMPL-GPTexture (skinned multi-person linear model - general purpose Texture), a novel pipeline that takes natural language prompts as input and leverages a state-of-the-art text-to-image generation model to produce paired high-resolution front and back images of a human subject as the starting point for texture estimation. Using the generated paired dual-view images, we first employ a human mesh recovery model to obtain a robust 2D-to-3D SMPL alignment between image pixels and the 3D model's UV coordinates for each views. Second, we use an inverted rasterization technique that explicitly projects the observed colour from the input images into the UV space, thereby producing accurate, complete texture maps. Finally, we apply a diffusion-based inpainting module to fill in the missing regions, and the fusion mechanism then combines these results into a unified full texture map. Extensive experiments shows that our SMPL-GPTexture can generate high resolution texture aligned with user's prompts.
Abstract (translated)
生成高质量、逼真的3D人体头像纹理对于计算机视觉和多媒体领域而言仍然是一个基础而具有挑战性的任务。然而，由于隐私、伦理以及获取成本的限制，真实的人体前后配对图像非常难以获得，这在很大程度上限制了数据集的规模。此外，使用深度生成模型（如GAN或扩散模型）从图像输入中学习先验知识以推断未见过的区域（例如人体背面），往往会导致伪影、结构不一致或细节损失等问题。 为了解决这些问题，我们提出了SMPL-GPTexture（具有通用纹理的多个人体线性模型），这是一种创新的工作流程。该工作流程使用自然语言提示作为输入，并利用最先进的文本到图像生成模型来创建配对的高分辨率前后人体视图图像，以用作纹理估计的基础。通过使用生成的双视角图像，我们首先采用一个人体网格恢复模型，以获得在每个视图中将图像像素与3D模型UV坐标进行可靠2D至3D SMPL对齐的方法。其次，我们利用一种逆向光栅化技术，该技术明确地将输入图像中的观察颜色投影到UV空间中，从而生成准确、完整的纹理贴图。最后，我们采用基于扩散的填补模块来填充缺失区域，并使用融合机制将这些结果结合成一个统一的整体纹理贴图。 广泛的实验表明，我们的SMPL-GPTexture能够根据用户的提示生成高质量并与其描述相匹配的高分辨率纹理。
URL
https://arxiv.org/abs/2504.13378
