Abstract
Conditional diffusion models have exhibited superior performance in high-fidelity text-guided visual generation and editing. Nevertheless, prevailing text-guided visual diffusion models primarily focus on incorporating text-visual relationships exclusively into the reverse process, often disregarding their relevance in the forward process. This inconsistency between forward and reverse processes may limit the precise conveyance of textual semantics in visual synthesis results. To address this issue, we propose a novel and general contextualized diffusion model (ContextDiff) by incorporating the cross-modal context encompassing interactions and alignments between text condition and visual sample into forward and reverse processes. We propagate this context to all timesteps in the two processes to adapt their trajectories, thereby facilitating cross-modal conditional modeling. We generalize our contextualized diffusion to both DDPMs and DDIMs with theoretical derivations, and demonstrate the effectiveness of our model in evaluations with two challenging tasks: text-to-image generation, and text-to-video editing. In each task, our ContextDiff achieves new state-of-the-art performance, significantly enhancing the semantic alignment between text condition and generated samples, as evidenced by quantitative and qualitative evaluations. Our code is available at this https URL
Abstract (translated)
条件扩散模型在高端文本指导下的视觉生成和编辑方面表现出了卓越的性能。然而，当前主要关注于将文本-视觉关系仅引入反向过程，而忽视其在正向过程的 relevance。这种正向和反向过程之间的不一致性可能限制了文本合成结果中精确传达文本语义的能力。为了解决这个问题，我们提出了一个新颖且一般化的条件扩散模型（ContextDiff），通过将跨模态上下文涵盖文本条件和视觉样本之间的交互和匹配引入到正向和反向过程中，从而实现文本条件下的扩散。我们将这个上下文传递到两个过程的所有时间步，以适应它们的轨迹，从而促进跨模态条件建模。我们对DDPM和DDIMs进行了理论推导，并展示了我们的模型在两个具有挑战性的任务上的效果：文本到图像生成和文本到视频编辑。在每项任务中，我们的ContextDiff都实现了最先进的性能，显著增强了文本条件和生成样本之间的语义对齐，正如定量和定性评估所证明的。我们的代码可以从该链接下载：
URL
https://arxiv.org/abs/2402.16627
