Abstract
The conditional text-to-image diffusion models have garnered significant attention in recent years. However, the precision of these models is often compromised mainly for two reasons, ambiguous condition input and inadequate condition guidance over single denoising loss. To address the challenges, we introduce two innovative solutions. Firstly, we propose a Spatial Guidance Injector (SGI) which enhances conditional detail by encoding text inputs with precise annotation information. This method directly tackles the issue of ambiguous control inputs by providing clear, annotated guidance to the model. Secondly, to overcome the issue of limited conditional supervision, we introduce Diffusion Consistency Loss (DCL), which applies supervision on the denoised latent code at any given time step. This encourages consistency between the latent code at each time step and the input signal, thereby enhancing the robustness and accuracy of the output. The combination of SGI and DCL results in our Effective Controllable Network (ECNet), which offers a more accurate controllable end-to-end text-to-image generation framework with a more precise conditioning input and stronger controllable supervision. We validate our approach through extensive experiments on generation under various conditions, such as human body skeletons, facial landmarks, and sketches of general objects. The results consistently demonstrate that our method significantly enhances the controllability and robustness of the generated images, outperforming existing state-of-the-art controllable text-to-image models.
Abstract (translated)
近年来，条件文本到图像扩散模型已经引起了广泛的关注。然而，这些模型的精度往往因为两个主要原因而妥协，即模糊的条件输入和单向去噪损失中的条件指导不足。为了应对这些挑战，我们提出了两种创新解决方案。首先，我们提出了一种空间引导器（SGI），通过为文本输入提供精确的注释信息来增强条件细节。这种方法直接解决了模糊控制输入的问题，并为模型提供了明确的、注释的指导。其次，为了克服有限的条件监督问题，我们引入了扩散一致损失（DCL），对任意给定时间点的去噪潜在码进行监督。这鼓励了每个时间步的潜在码与输入信号之间的一致性，从而提高了输出的稳健性和准确性。SGI和DCL的结合我们产生了有效的可控制网络（ECNet），它具有更精确的控制输入和更强的可控制监督，从而实现了更准确、更可靠的文本到图像生成。我们通过在各种条件下进行广泛的实验来验证我们的方法。实验结果一致表明，我们的方法显著增强了生成图像的可控性和稳健性，超过了现有最先进的可控制文本到图像模型。
URL
https://arxiv.org/abs/2403.18417
