Abstract
Inpainting, the process of filling missing or corrupted image parts, has broad applications, including medical imaging. However, in specialized fields like medical polyps imaging, where accuracy and reliability are critical, inpainting models can generate inaccurate images, leading to significant errors in medical diagnosis and treatment. To ensure reliability, medical images should be annotated by experts like oncologists for effective model training. We propose PrefPaint, an approach that incorporates human feedback into the training process of Stable Diffusion Inpainting, bypassing the need for computationally expensive reward models. In addition, we develop a web-based interface streamlines training, fine-tuning, and inference. This interactive interface provides a smooth and intuitive user experience, making it easier to offer feedback and manage the fine-tuning process. User study on various domains shows that PrefPaint outperforms existing methods, reducing visual inconsistencies and improving image rendering, particularly in medical contexts, where our model generates more realistic polyps images.
Abstract (translated)
图像修复（Inpainting），即填充缺失或损坏的图像部分，具有广泛的应用领域，包括医学成像。然而，在如医疗息肉成像这样对准确性和可靠性要求极高的专业领域中，图像修复模型可能会生成不准确的图像，这会导致严重的误诊和治疗错误。为了确保医学图像的可靠性和准确性，应该由放射科医生等专家对其进行标注，以进行有效的模型训练。 我们提出了一种名为PrefPaint的方法，该方法将人类反馈整合到Stable Diffusion Inpainting（稳定扩散修复）模型的训练过程中，并且无需使用计算成本高昂的奖励模型。此外，我们还开发了一个基于网络的界面，用于简化训练、微调和推理过程。这个交互式界面提供了流畅直观的用户体验，使得提供反馈和管理微调过程变得更加容易。 在多个领域的用户研究中显示，PrefPaint方法超越了现有的技术，在减少视觉不一致性和提高图像渲染质量方面表现出色，特别是在医学背景下，我们的模型能够生成更加逼真的息肉图像。
URL
https://arxiv.org/abs/2506.21834
