Abstract
Text-to-image diffusion models (T2I) use a latent representation of a text prompt to guide the image generation process. However, the process by which the encoder produces the text representation is unknown. We propose the Diffusion Lens, a method for analyzing the text encoder of T2I models by generating images from its intermediate representations. Using the Diffusion Lens, we perform an extensive analysis of two recent T2I models. Exploring compound prompts, we find that complex scenes describing multiple objects are composed progressively and more slowly compared to simple scenes; Exploring knowledge retrieval, we find that representation of uncommon concepts requires further computation compared to common concepts, and that knowledge retrieval is gradual across layers. Overall, our findings provide valuable insights into the text encoder component in T2I pipelines.
Abstract (translated)
文本到图像扩散模型（T2I）使用文本提示的潜在表示来指导图像生成过程。然而，编码器产生文本表示的过程是未知的。我们提出了一种名为扩散镜头的方法，通过从其中间表示生成图像来分析T2I模型的文本编码器。使用扩散镜头，我们对两个最近的T2I模型进行了广泛的分析。探索组合提示，我们发现复杂场景描述多个对象的逐渐和缓慢组成与简单场景相比；探索知识检索，我们发现与常见概念相比，不寻常概念的表示需要进一步计算，并且知识检索在层次结构中是逐渐发展的。总体而言，我们的研究结果为T2I模型的文本编码器部件提供了宝贵的洞见。
URL
https://arxiv.org/abs/2403.05846
