Abstract
Diffusion models show a remarkable ability in generating images that closely mirror the training distribution. However, these models are prone to training data memorization, leading to significant privacy, ethical, and legal concerns, particularly in sensitive fields such as medical imaging. We hypothesize that memorization is driven by the overparameterization of deep models, suggesting that regularizing model capacity during fine-tuning could be an effective mitigation strategy. Parameter-efficient fine-tuning (PEFT) methods offer a promising approach to capacity control by selectively updating specific parameters. However, finding the optimal subset of learnable parameters that balances generation quality and memorization remains elusive. To address this challenge, we propose a bi-level optimization framework that guides automated parameter selection by utilizing memorization and generation quality metrics as rewards. Our framework successfully identifies the optimal parameter set to be updated to satisfy the generation-memorization tradeoff. We perform our experiments for the specific task of medical image generation and outperform existing state-of-the-art training-time mitigation strategies by fine-tuning as few as 0.019% of model parameters. Furthermore, we show that the strategies learned through our framework are transferable across different datasets and domains. Our proposed framework is scalable to large datasets and agnostic to the choice of reward functions. Finally, we show that our framework can be combined with existing approaches for further memorization mitigation.
Abstract (translated)
扩散模型在生成与训练分布非常相似的图像方面表现出出色的能力。然而，这些模型容易受到训练数据记忆的影响，导致隐私、道德和法律问题，特别是在敏感领域（如医学成像）更是如此。我们假设，记忆是由深度模型的过参数化引起的，因此通过在微调过程中对模型能力进行正则化可能是一种有效的减轻策略。参数高效的微调（PEFT）方法提供了一个有前景的方法，通过选择性地更新特定参数来控制模型容量。然而，找到平衡生成质量和记忆的最佳可学习参数集仍然具有挑战性。为解决这个问题，我们提出了一个二层优化框架，通过使用记忆和生成质量度量作为奖励来指导自动参数选择。我们的框架成功找到了要更新的最佳参数集以满足生成-记忆权衡。我们对特定任务的医学图像生成进行实验，并比现有的训练时间减轻策略提高了约0.019%的模型参数。此外，我们还证明了通过我们的框架学习到的策略在不同数据集和领域上是可转移的。我们的框架具有可扩展到大型数据集的规模，对奖励函数的选择不敏感。最后，我们还展示了我们的框架可以与现有的记忆减轻方法相结合，进一步减轻记忆的影响。
URL
https://arxiv.org/abs/2405.19458
