Abstract
This paper investigates image inpainting with preference alignment. Instead of introducing a novel method, we go back to basics and revisit fundamental problems in achieving such alignment. We leverage the prominent direct preference optimization approach for alignment training and employ public reward models to construct preference training datasets. Experiments are conducted across nine reward models, two benchmarks, and two baseline models with varying structures and generative algorithms. Our key findings are as follows: (1) Most reward models deliver valid reward scores for constructing preference data, even if some of them are not reliable evaluators. (2) Preference data demonstrates robust trends in both candidate scaling and sample scaling across models and benchmarks. (3) Observable biases in reward models, particularly in brightness, composition, and color scheme, render them susceptible to cause reward hacking. (4) A simple ensemble of these models yields robust and generalizable results by mitigating such biases. Built upon these observations, our alignment models significantly outperform prior models across standard metrics, GPT-4 assessments, and human evaluations, without any changes to model structures or the use of new datasets. We hope our work can set a simple yet solid baseline, pushing this promising frontier. Our code is open-sourced at: this https URL.
Abstract (translated)
本文探讨了基于偏好对齐的图像修复技术。不同于提出一种新的方法，我们回归基础，重新审视实现这种对齐的基本问题。我们利用突出的直接偏好优化方法进行对齐训练，并使用公开的奖励模型来构建偏好数据集。实验在九种不同的奖励模型、两个基准测试以及两种结构和生成算法各不相同的基线模型上进行。我们的主要发现如下：（1）大多数奖励模型可以为构造偏好数据提供有效的评分，即使其中一些并不是可靠的评估工具。（2）偏好数据在不同候选方案及样本数量上的趋势对所有模型和基准测试来说都非常稳定。（3）奖励模型中存在的可观察偏差，尤其是亮度、构图和色彩方案方面的偏差，使得它们容易受到奖励作弊行为的影响。（4）这些模型的简单集成可以缓解这种偏差，并产生稳健且通用的结果。基于以上发现，我们的对齐模型在标准度量、GPT-4评估以及人工评价方面均显著超越了先前的模型，而无需更改任何模型结构或引入新的数据集。我们希望本项工作能够为该领域设立一个简单而坚实的基准，推动这一充满前景的研究前沿的发展。我们的代码可在以下网址开放获取：[此 HTTPS URL]。
URL
https://arxiv.org/abs/2509.23082
