Abstract
We introduce novel methods for adapting diffusion models under differential privacy (DP) constraints, enabling privacy-preserving style and content transfer without fine-tuning. Traditional approaches to private adaptation, such as DP-SGD, incur significant computational overhead and degrade model performance when applied to large, complex models. Our approach instead leverages embedding-based techniques: Universal Guidance and Textual Inversion (TI), adapted with differentially private mechanisms. We apply these methods to Stable Diffusion for style adaptation using two private datasets: a collection of artworks by a single artist and pictograms from the Paris 2024 Olympics. Experimental results show that the TI-based adaptation achieves superior fidelity in style transfer, even under strong privacy guarantees, while both methods maintain high privacy resilience by employing calibrated noise and subsampling strategies. Our findings demonstrate a feasible and efficient pathway for privacy-preserving diffusion model adaptation, balancing data protection with the fidelity of generated images, and offer insights into embedding-driven methods for DP in generative AI applications.
Abstract (translated)
我们介绍了一种在差分隐私（DP）约束下适应扩散模型的新方法，该方法能够实现在无需微调的情况下进行保护隐私的风格和内容迁移。传统的私有化适应方法如DP-SGD，在应用于大型复杂模型时会导致显著的计算开销并降低模型性能。而我们的方法则利用基于嵌入的技术：通用指导和文本反转（TI），并通过差分隐私机制对其进行调整。我们将这些方法应用于Stable Diffusion，使用两个私有数据集进行风格适应实验：一位艺术家的作品集合以及2024年巴黎奥运会的图标。实验结果表明，在强有力的隐私保证下，基于TI的方法在风格迁移中达到了更高的保真度；同时，两种方法都通过校准噪声和子采样策略保持了较高的隐私韧性。我们的研究发现提供了一条可行且高效的途径，实现了扩散模型适应过程中的数据保护与生成图像保真度之间的平衡，并为嵌入驱动的差分隐私技术在生成式AI应用中提供了见解。
URL
https://arxiv.org/abs/2411.14639
