Abstract
3D human generation is an important problem with a wide range of applications in computer vision and graphics. Despite recent progress in generative AI such as diffusion models or rendering methods like Neural Radiance Fields or Gaussian Splatting, controlling the generation of accurate 3D humans from text prompts remains an open challenge. Current methods struggle with fine detail, accurate rendering of hands and faces, human realism, and controlability over appearance. The lack of diversity, realism, and annotation in human image data also remains a challenge, hindering the development of a foundational 3D human model. We present a weakly supervised pipeline that tries to address these challenges. In the first step, we generate a photorealistic human image dataset with controllable attributes such as appearance, race, gender, etc using a state-of-the-art image diffusion model. Next, we propose an efficient mapping approach from image features to 3D point clouds using a transformer-based architecture. Finally, we close the loop by training a point-cloud diffusion model that is conditioned on the same text prompts used to generate the original samples. We demonstrate orders-of-magnitude speed-ups in 3D human generation compared to the state-of-the-art approaches, along with significantly improved text-prompt alignment, realism, and rendering quality. We will make the code and dataset available.
Abstract (translated)
三维人体生成是计算机视觉和图形学中一个具有广泛应用的重要问题。尽管在生成式AI（如扩散模型）以及渲染方法（例如神经辐射场或高斯点聚法）方面取得了近期进展，但从文本提示准确生成三维人体仍然是一个开放性的挑战。当前的方法在精细细节、手部和面部的精确渲染、人类的真实感以及外观控制性上存在困难。此外，由于人体图像数据中缺乏多样性、真实性和标注信息，开发基础的人体三维模型也面临阻碍。 我们提出了一种弱监督管道来解决这些挑战。首先，利用最先进的图像扩散模型生成包含可控属性（如外观、种族、性别等）的逼真人体图像数据集。接下来，我们提出了一个基于变压器架构的有效映射方法，用于将图像特征转换为三维点云。最后，通过训练条件相同的文本提示以生成原始样本为基础的点云扩散模型来闭合这一循环。 我们的方法在三维人体生成速度上相比现有最佳方法提高了几个数量级，并显著改善了文本提示的对齐性、真实感和渲染质量。我们将公开发布代码和数据集。
URL
https://arxiv.org/abs/2506.04351
