Abstract
The rapid evolution of text-to-image diffusion models has opened the door of generative AI, enabling the translation of textual descriptions into visually compelling images with remarkable quality. However, a persistent challenge within this domain is the optimization of prompts to effectively convey abstract concepts into concrete objects. For example, text encoders can hardly express "peace", while can easily illustrate olive branches and white doves. This paper introduces a novel approach named Prompt Optimizer for Abstract Concepts (POAC) specifically designed to enhance the performance of text-to-image diffusion models in interpreting and generating images from abstract concepts. We propose a Prompt Language Model (PLM), which is initialized from a pre-trained language model, and then fine-tuned with a curated dataset of abstract concept prompts. The dataset is created with GPT-4 to extend the abstract concept to a scene and concrete objects. Our framework employs a Reinforcement Learning (RL)-based optimization strategy, focusing on the alignment between the generated images by a stable diffusion model and optimized prompts. Through extensive experiments, we demonstrate that our proposed POAC significantly improves the accuracy and aesthetic quality of generated images, particularly in the description of abstract concepts and alignment with optimized prompts. We also present a comprehensive analysis of our model's performance across diffusion models under different settings, showcasing its versatility and effectiveness in enhancing abstract concept representation.
Abstract (translated)
文本到图像扩散模型的快速演变为生成型人工智能打开了大门，使将文本描述转化为具有引人入胜视觉效果的图像成为可能，特别是在描述抽象概念方面。然而，这一领域的一个持续挑战是优化提示以有效地传达抽象概念为具体物体。例如，文本编码器很难表达“和平”，但可以轻松地描绘橄榄枝和白鸽子。本文介绍了一种名为抽象概念提示优化器（POAC）的新颖方法，专门设计用于提高文本到图像扩散模型在解释和生成图像时的性能。我们提出了一个基于预训练语言模型的Prompt语言模型（PLM），然后用经过精心挑选的抽象概念提示数据集进行微调。数据集使用GPT-4来扩展抽象概念场景和物体。我们的框架采用了一种基于强化学习的优化策略，重点关注通过稳定的扩散模型生成的图像与优化提示之间的对齐。通过广泛的实验，我们证明了我们的POAC显著提高了生成图像的准确性和美学质量，特别是在描述抽象概念和与优化提示对齐方面。我们还对在不同设置下的扩散模型性能进行了全面的分析，展示了模型在增强抽象概念表示方面的多样性和有效性。
URL
https://arxiv.org/abs/2404.11589
