Abstract
Generative models have been shown to "memorize" certain training data, leading to verbatim or near-verbatim generating images, which may cause privacy concerns or copyright infringement. We introduce Guidance Using Attractive-Repulsive Dynamics (GUARD), a novel framework for memorization mitigation in text-to-image diffusion models. GUARD adjusts the image denoising process to guide the generation away from an original training image and towards one that is distinct from training data while remaining aligned with the prompt, guarding against reproducing training data, without hurting image generation quality. We propose a concrete instantiation of this framework, where the positive target that we steer towards is given by a novel method for (cross) attention attenuation based on (i) a novel statistical mechanism that automatically identifies the prompt positions where cross attention must be attenuated and (ii) attenuating cross-attention in these per-prompt locations. The resulting GUARD offers a surgical, dynamic per-prompt inference-time approach that, we find, is by far the most robust method in terms of consistently producing state-of-the-art results for memorization mitigation across two architectures and for both verbatim and template memorization, while also improving upon or yielding comparable results in terms of image quality.
Abstract (translated)
生成模型已被证明会“记忆”某些训练数据，导致生成图像时出现文字上或近似于原文的内容，这可能会引发隐私问题或版权侵权。为此，我们提出了一个新框架——利用吸引-排斥动力学的指引（Guidance Using Attractive-Repulsive Dynamics, GUARD），用于减轻文本到图像扩散模型中的记忆问题。GUARD通过调整图像去噪过程来引导生成结果远离原始训练图像，并使其与提示保持一致的同时与其他训练数据区分开来，从而防止再现训练数据，而不会影响图像的生成质量。 我们提出了一种具体的框架实现方式：其中正向目标由一种基于（i）一种新统计机制自动识别需要减弱交叉注意力的提示位置以及(ii) 在这些特定提示位置上减弱交叉注意力的新方法给出。由此产生的GUARD提供了一种手术般的、动态的提示时间推理方法，我们发现这种方法在两个架构中对于文字和模板记忆的消除方面表现出了迄今为止最为稳健的方法，并且在图像质量方面也有所改进或达到了可比的结果。
URL
https://arxiv.org/abs/2603.00133
