Abstract
Foreground-conditioned inpainting, which aims at generating a harmonious background for a given foreground subject based on the text prompt, is an important subfield in controllable image generation. A common challenge in current methods, however, is the occurrence of Spatial Relationship Hallucinations between the foreground subject and the generated background, including inappropriate scale, positional relationships, and viewpoints. Critically, the subjective nature of spatial rationality makes it challenging to quantify, hindering the use of traditional reward-based RLHF methods. To address this issue, we propose InpaintDPO, the first Direct Preference Optimization (DPO) based framework dedicated to spatial rationality in foreground-conditioned inpainting, ensuring plausible spatial relationships between foreground and background elements. To resolve the gradient conflicts in standard DPO caused by identical foreground in win-lose pairs, we propose MaskDPO, which confines preference optimization exclusively to the background to enhance background spatial relationships, while retaining the inpainting loss in the foreground region for robust foreground preservation. To enhance coherence at the foreground-background boundary, we propose Conditional Asymmetric Preference Optimization, which samples pairs with differentiated cropping operations and applies global preference optimization to promote contextual awareness and enhance boundary coherence. Finally, based on the observation that winning samples share a commonality in plausible spatial relationships, we propose Shared Commonality Preference Optimization to enhance the model's understanding of spatial commonality across high-quality winning samples, further promoting shared spatial rationality.
Abstract (translated)
前景条件下的图像修复（Foreground-conditioned inpainting）旨在根据文本提示生成与给定前景主体相协调的背景，这是可控图像生成领域的一个重要子领域。然而，当前方法面临的一个常见挑战是前景主体和生成背景之间会出现空间关系幻觉问题，包括不适当的尺度、位置关系以及视角问题。特别地，空间合理性的主观性使其难以量化，从而阻碍了传统基于奖励的RLHF（Reinforcement Learning from Human Feedback）方法的应用。 为解决这一问题，我们提出了InpaintDPO——第一个专注于前景条件修复中空间合理性问题的直接偏好优化(Direct Preference Optimization, DPO)框架，以确保前景和背景元素之间有合理的空间关系。为了缓解标准DPO在胜者与败者配对时由于前景相同而造成的梯度冲突问题，我们提出了MaskDPO方法，该方法将偏好优化严格限制在背景部分以增强背景的空间关系，同时保留前景区域的修复损失以确保稳固地保持前景内容。 为提高前景和背景边界处的一致性，我们提出了一种条件非对称偏好优化（Conditional Asymmetric Preference Optimization）方法。这种方法通过采用不同的裁剪操作来采样配对样本，并应用全局偏好优化以增强上下文感知能力并改善边界一致性。 最后，基于观察到获胜的样本在合理的空间关系上具有共同点这一发现，我们提出了共享共性偏奷新框架（Shared Commonality Preference Optimization）。该方法通过分析高质量的胜利样本之间的空间共性来提高模型对空间普遍性的理解，并进一步推广这种共享的空间合理性。
URL
https://arxiv.org/abs/2512.15644
