Abstract
Recent studies have demonstrated the exceptional potentials of leveraging human preference datasets to refine text-to-image generative models, enhancing the alignment between generated images and textual prompts. Despite these advances, current human preference datasets are either prohibitively expensive to construct or suffer from a lack of diversity in preference dimensions, resulting in limited applicability for instruction tuning in open-source text-to-image generative models and hinder further exploration. To address these challenges and promote the alignment of generative models through instruction tuning, we leverage multimodal large language models to create VisionPrefer, a high-quality and fine-grained preference dataset that captures multiple preference aspects. We aggregate feedback from AI annotators across four aspects: prompt-following, aesthetic, fidelity, and harmlessness to construct VisionPrefer. To validate the effectiveness of VisionPrefer, we train a reward model VP-Score over VisionPrefer to guide the training of text-to-image generative models and the preference prediction accuracy of VP-Score is comparable to human annotators. Furthermore, we use two reinforcement learning methods to supervised fine-tune generative models to evaluate the performance of VisionPrefer, and extensive experimental results demonstrate that VisionPrefer significantly improves text-image alignment in compositional image generation across diverse aspects, e.g., aesthetic, and generalizes better than previous human-preference metrics across various image distributions. Moreover, VisionPrefer indicates that the integration of AI-generated synthetic data as a supervisory signal is a promising avenue for achieving improved alignment with human preferences in vision generative models.
Abstract (translated)
近年来，研究表明，利用人类偏好数据集来优化文本到图像生成模型具有异常的潜力，从而增强生成图像与文本提示之间的对齐。尽管取得了这些进步，但目前的用户偏好数据集要么过于昂贵，难以构建，要么在偏好维度上缺乏多样性，导致对于开源文本到图像生成模型的指令调整应用有限。为了应对这些挑战，促进生成模型的指令调整以实现更好的人机偏好匹配，我们利用多模态大型语言模型创建了VisionPrefer，一个高质量且细粒度的偏好数据集，涵盖了多个偏好方面。我们通过汇总AI注释者的反馈，从提示跟随、美学、忠实度和无害性四个方面构建VisionPrefer。为了验证VisionPrefer的有效性，我们在VisionPrefer上训练了一个奖励模型VP-Score，以指导文本到图像生成模型的训练和VP-Score的偏好预测精度与人类注释者相当。此外，我们还使用两种强化学习方法，在监督下微调生成模型以评估VisionPrefer的表现，实验结果表明，VisionPrefer在各种图像分布下的文本图像对齐方面显著改善，例如美学和扩展性优于以前的人类偏好指标。此外，VisionPrefer表明，将AI生成的模拟数据作为监督信号实现人机偏好与视觉生成模型的对齐是一个有前景的方向。
URL
https://arxiv.org/abs/2404.15100
