Abstract
Despite significant recent advances in image generation with diffusion models, their internal latent representations remain poorly understood. Existing works focus on the bottleneck layer (h-space) of Stable Diffusion's U-Net or leverage the cross-attention, self-attention, or decoding layers. Our model, SkipInject takes advantage of U-Net's skip connections. We conduct thorough analyses on the role of the skip connections and find that the residual connections passed by the third encoder block carry most of the spatial information of the reconstructed image, splitting the content from the style. We show that injecting the representations from this block can be used for text-based editing, precise modifications, and style transfer. We compare our methods state-of-the-art style transfer and image editing methods and demonstrate that our method obtains the best content alignment and optimal structural preservation tradeoff.
Abstract (translated)
尽管基于扩散模型的图像生成技术取得了重大进展，但对其内部潜在表示的理解仍然不足。现有的研究主要集中在稳定扩散模型U-Net中的瓶颈层（h-space）上，或者利用交叉注意力、自我注意或解码层。我们的模型SkipInject则充分利用了U-Net的跳跃连接。我们对跳跃连接的作用进行了深入分析，并发现由第三编码块传递的残差连接携带了重建图像中大部分的空间信息，从而将内容与风格分离。我们展示了从该区块注入表示的方法可以用于基于文本的编辑、精确修改和风格迁移。 我们将我们的方法与最先进的风格迁移和图像编辑方法进行比较，并证明我们的方法在内容对齐和结构保存的最佳权衡方面表现最佳。
URL
https://arxiv.org/abs/2501.14524
