Abstract
Recent text-to-image generative models have exhibited remarkable abilities in generating high-fidelity and photo-realistic images. However, despite the visually impressive results, these models often struggle to preserve plausible human structure in the generations. Due to this reason, while generative models have shown promising results in aiding downstream image recognition tasks by generating large volumes of synthetic data, they remain infeasible for improving downstream human pose perception and understanding. In this work, we propose Diffusion model with Human Pose Correction (Diffusion HPC), a text-conditioned method that generates photo-realistic images with plausible posed humans by injecting prior knowledge about human body structure. We show that Diffusion HPC effectively improves the realism of human generations. Furthermore, as the generations are accompanied by 3D meshes that serve as ground truths, Diffusion HPC's generated image-mesh pairs are well-suited for downstream human mesh recovery task, where a shortage of 3D training data has long been an issue.
Abstract (translated)
最近,文本生成图像生成模型表现出惊人的能力,能够在生成高保真的、照片般的图像方面实现。然而,尽管这些模型取得了令人瞩目的成果,但它们在生成代中保留合理人类结构方面往往面临困难。因此,虽然生成模型通过生成大量合成数据在帮助后续图像识别任务方面表现出良好的结果,但它们仍然无法改善后续人类姿态感知和理解任务。在这项工作中,我们提出了一种人类姿态纠正扩散模型(扩散 HPC),这是一种基于文本的方法,通过注入人类身体结构的先验知识,生成具有合理人类姿态的图像。我们表明,扩散 HPC有效地改善了人类生成器的逼真度。此外,由于生成代伴随着3D网格作为基准现实,扩散 HPC生成的图像网格对后续人类网格恢复任务非常适用,而长期存在的3D训练数据短缺问题已经导致了这个问题。
URL
https://arxiv.org/abs/2303.09541
