Abstract
In this paper, we introduce a Key-point-guided Diffusion probabilistic Model (KDM) that gains precise control over images by manipulating the object's key-point. We propose a two-stage generative model incorporating an optical flow map as an intermediate output. By doing so, a dense pixel-wise understanding of the semantic relation between the image and sparse key point is configured, leading to more realistic image generation. Additionally, the integration of optical flow helps regulate the inter-frame variance of sequential images, demonstrating an authentic sequential image generation. The KDM is evaluated with diverse key-point conditioned image synthesis tasks, including facial image generation, human pose synthesis, and echocardiography video prediction, demonstrating the KDM is proving consistency enhanced and photo-realistic images compared with state-of-the-art models.
Abstract (translated)
在本文中,我们提出了一种名为Key-point-guided Diffusion probabilistic Model(KDM)的模型,通过操纵对象的 key-point来精确控制图像。我们提出了一种包含光学流图的中间输出两级生成模型。通过这样做,我们获得了对图像中视觉关系进行全面理解,从而实现了更逼真的图像生成。此外,光流图的整合有助于调节连续图像之间的帧间方差,证明了KDM具有与最先进模型相同的真实感和照片写实感。KDM通过各种条件下的键点约束图像合成任务进行了评估,包括面部图像生成、人体姿态合成和超声心动图视频预测,证明了KDM与最先进模型的 consistency得到了增强,同时保持了照片写实感。
URL
https://arxiv.org/abs/2401.08178