Abstract
The malformed hands in the AI-generated images seriously affect the authenticity of the images. To refine malformed hands, existing depth-based approaches use a hand depth estimator to guide the refinement of malformed hands. Due to the performance limitations of the hand depth estimator, many hand details cannot be represented, resulting in errors in the generated hands, such as confusing the palm and the back of the hand. To solve this problem, we propose a 3D mesh-guided refinement framework using a diffusion pipeline. We use a state-of-the-art 3D hand mesh estimator, which provides more details of the hands. For training, we collect and reannotate a dataset consisting of RGB images and 3D hand mesh. Then we design a diffusion inpainting model to generate refined outputs guided by 3D hand meshes. For inference, we propose a double check algorithm to facilitate the 3D hand mesh estimator to obtain robust hand mesh guidance to obtain our refined results. Beyond malformed hand refinement, we propose a novel hand pose transformation method. It increases the flexibility and diversity of the malformed hand refinement task. We made the restored images mimic the hand poses of the reference images. The pose transformation requires no additional training. Extensive experimental results demonstrate the superior performance of our proposed method.
Abstract (translated)
在AI生成的图像中，畸形的手严重影响了图片的真实性。现有的基于深度的方法通过使用手部深度估计器来指导畸形手部的精炼过程。然而，由于手部深度估计器性能的限制，许多手部细节无法被准确表示，导致生成的手部出现错误，例如将手掌与手背混淆。为了解决这个问题，我们提出了一种基于3D网格引导的精炼框架，并采用扩散管道进行优化。我们使用最先进的3D手部网格估计算法，该算法提供了更多的手部细节信息。 在训练阶段，我们收集并重新标注了一个包含RGB图像和3D手部网格的数据集。然后设计一个基于扩散的修补模型，以生成由3D手部网格引导的精炼输出结果。在推断过程中，我们提出了一种双检查算法，使3D手部网格估计算法能够获得稳健的手部网格指引，从而得到我们的精炼结果。 除了畸形手部的精炼外，我们还提出了一个新颖的手势变换方法，这种方法增加了畸形手部精炼任务的灵活性和多样性。我们将恢复后的图像模仿参考图像中的手势动作，而这种姿态转换并不需要额外的训练过程。 通过广泛的实验结果证明了我们所提出的方法具有卓越性能。
URL
https://arxiv.org/abs/2506.12680
