Abstract
We introduce a novel diffusion transformer, LazyDiffusion, that generates partial image updates efficiently. Our approach targets interactive image editing applications in which, starting from a blank canvas or an image, a user specifies a sequence of localized image modifications using binary masks and text prompts. Our generator operates in two phases. First, a context encoder processes the current canvas and user mask to produce a compact global context tailored to the region to generate. Second, conditioned on this context, a diffusion-based transformer decoder synthesizes the masked pixels in a "lazy" fashion, i.e., it only generates the masked region. This contrasts with previous works that either regenerate the full canvas, wasting time and computation, or confine processing to a tight rectangular crop around the mask, ignoring the global image context altogether. Our decoder's runtime scales with the mask size, which is typically small, while our encoder introduces negligible overhead. We demonstrate that our approach is competitive with state-of-the-art inpainting methods in terms of quality and fidelity while providing a 10x speedup for typical user interactions, where the editing mask represents 10% of the image.
Abstract (translated)
我们提出了一种名为LazyDiffusion的新扩散变换器，它能够高效地生成部分图像更新。我们的方法针对交互式图像编辑应用，用户从空白的画布或图像开始，使用二进制掩码和文本提示指定一系列局部图像修改序列。生成器有两个阶段。首先，上下文编码器处理当前画布和用户掩码以生成一个紧凑的全局上下文，专门针对要生成的区域进行优化。其次，在上下文条件下，扩散基变换器解码器以“懒散”的方式合成掩码中的像素，即它只生成掩码的区域。这 contrasts with previous works that either regenerate the full canvas, wasting time and computation, or confine processing to a tight rectangular crop around the mask, ignoring the global image context altogether. 我们的解码器的运行时与掩码大小成比例，而我们的编码器引入的开销可以忽略不计。我们证明了我们的方法在质量和忠实度方面与最先进的修复方法相当，同时为典型的用户交互提供10倍的加速，其中编辑掩码代表图像的10%。
URL
https://arxiv.org/abs/2404.12382
