Abstract
While text-to-image generation has been extensively studied, generating images from scene graphs remains relatively underexplored, primarily due to challenges in accurately modeling spatial relationships and object interactions. To fill this gap, we introduce Scene-Bench, a comprehensive benchmark designed to evaluate and enhance the factual consistency in generating natural scenes. Scene-Bench comprises MegaSG, a large-scale dataset of one million images annotated with scene graphs, facilitating the training and fair comparison of models across diverse and complex scenes. Additionally, we propose SGScore, a novel evaluation metric that leverages chain-of-thought reasoning capabilities of multimodal large language models (LLMs) to assess both object presence and relationship accuracy, offering a more effective measure of factual consistency than traditional metrics like FID and CLIPScore. Building upon this evaluation framework, we develop a scene graph feedback pipeline that iteratively refines generated images by identifying and correcting discrepancies between the scene graph and the image. Extensive experiments demonstrate that Scene-Bench provides a more comprehensive and effective evaluation framework compared to existing benchmarks, particularly for complex scene generation. Furthermore, our feedback strategy significantly enhances the factual consistency of image generation models, advancing the field of controllable image generation.
Abstract (translated)
尽管文本到图像的生成已经得到了广泛的研究，但从场景图生成图像仍然相对较少被探索，主要是因为准确建模空间关系和对象互动存在挑战。为了填补这一空白，我们引入了Scene-Bench，这是一个旨在评估并提高自然场景生成中事实一致性的全面基准测试。Scene-Bench 包含 MegaSG，一个由一百万张带有场景图注释的大型数据集，有助于在多样且复杂的场景中训练和公平比较模型。此外，我们提出了 SGScore，这是一种新颖的评价指标，利用多模态大语言模型（LLMs）的链式思考推理能力来评估对象存在及其关系的准确性，相比传统的FID和CLIPScore等度量标准，提供了更有效的事实一致性衡量方法。基于这一评估框架，我们开发了一个场景图反馈管道，通过识别并修正图像与场景图之间的差异来迭代地改进生成的图像。广泛的实验表明，Scene-Bench 提供了比现有基准测试更加全面且有效的评估框架，特别是在复杂场景生成方面。此外，我们的反馈策略显著提升了图像生成模型的事实一致性，推动了可控图像生成领域的进步。
URL
https://arxiv.org/abs/2411.15435
