Abstract
Recent advances in diffusion models can generate high-quality and stunning images from text. However, multi-turn image generation, which is of high demand in real-world scenarios, still faces challenges in maintaining semantic consistency between images and texts, as well as contextual consistency of the same subject across multiple interactive turns. To address this issue, we introduce TheaterGen, a training-free framework that integrates large language models (LLMs) and text-to-image (T2I) models to provide the capability of multi-turn image generation. Within this framework, LLMs, acting as a "Screenwriter", engage in multi-turn interaction, generating and managing a standardized prompt book that encompasses prompts and layout designs for each character in the target image. Based on these, Theatergen generate a list of character images and extract guidance information, akin to the "Rehearsal". Subsequently, through incorporating the prompt book and guidance information into the reverse denoising process of T2I diffusion models, Theatergen generate the final image, as conducting the "Final Performance". With the effective management of prompt books and character images, TheaterGen significantly improves semantic and contextual consistency in synthesized images. Furthermore, we introduce a dedicated benchmark, CMIGBench (Consistent Multi-turn Image Generation Benchmark) with 8000 multi-turn instructions. Different from previous multi-turn benchmarks, CMIGBench does not define characters in advance. Both the tasks of story generation and multi-turn editing are included on CMIGBench for comprehensive evaluation. Extensive experimental results show that TheaterGen outperforms state-of-the-art methods significantly. It raises the performance bar of the cutting-edge Mini DALLE 3 model by 21% in average character-character similarity and 19% in average text-image similarity.
Abstract (translated)
近年来，扩散模型的进步可以从文本中生成高质量和令人惊叹的图像。然而，在现实场景中，多轮图像生成仍然面临着在图像和文本之间保持语义一致性以及跨多个交互轮次同一主题下保持上下文一致性的挑战。为解决这个问题，我们引入了TheaterGen，一个无需训练的框架，将大型语言模型（LLMs）和文本到图像（T2I）模型集成在一起，提供多轮图像生成的能力。在这个框架内，LLM充当一个“编剧”，参与多轮交互，生成和管理工作目标图像中每个角色的标准化提示和布局设计。基于这些，Theatergen生成角色图像和提取指导信息，类似于“排练”。接着，将提示书和指导信息融入T2I扩散模型的反向去噪过程，Theatergen生成最终图像，进行“最后表演”。通过有效管理提示书和角色图像，Theatergen在生成的图像中显著提高了语义和上下文一致性。此外，我们还引入了一个专门基准，CMIGBench（一致多轮图像生成基准），具有8000个多轮指令。与之前的多人多轮基准不同，CMIGBench没有预先定义角色。在CMIGBench中，故事生成和多轮编辑任务都进行全面的评估。广泛的实验结果表明，TheaterGen在性能上远胜于最先进的方法。它将先进的小型DALLE 3模型的性能提高了21%的单个角色角色相似度和19%的文本图像相似度。
URL
https://arxiv.org/abs/2404.18919
