Abstract
Generative models trained on sensitive image datasets risk memorizing and reproducing individual training examples, making strong privacy guarantees essential. While differential privacy (DP) provides a principled framework for such guarantees, standard DP finetuning (e.g., with DP-SGD) often results in severe degradation of image quality, particularly in high-frequency textures, due to the indiscriminate addition of noise across all model parameters. In this work, we propose a spectral DP framework based on the hypothesis that the most privacy-sensitive portions of an image are often low-frequency components in the wavelet space (e.g., facial features and object shapes) while high-frequency components are largely generic and public. Based on this hypothesis, we propose the following two-stage framework for DP image generation with coarse image intermediaries: (1) DP finetune an autoregressive spectral image tokenizer model on the low-resolution wavelet coefficients of the sensitive images, and (2) perform high-resolution upsampling using a publicly pretrained super-resolution model. By restricting the privacy budget to the global structures of the image in the first stage, and leveraging the post-processing property of DP for detail refinement, we achieve promising trade-offs between privacy and utility. Experiments on the MS-COCO and MM-CelebA-HQ datasets show that our method generates images with improved quality and style capture relative to other leading DP image frameworks.
Abstract (translated)
训练在敏感图像数据集上的生成模型存在记忆和再现个别训练样本的风险，因此需要强有力的隐私保障。虽然差分隐私（DP）为这些保证提供了一个原则性框架，但使用标准DP微调方法（例如DP-SGD），通常会导致图像质量严重下降，特别是在高频纹理方面，这是因为在所有模型参数中无差别地添加噪声所导致的。 在本文工作中，我们提出了一种基于以下假设的光谱差分隐私框架：图像中最敏感于隐私的部分通常是小波空间中的低频成分（如面部特征和物体形状），而高频部分则大多是通用且公开的。根据这一假设，我们提出了一个两阶段框架用于具有粗略图像中间体的DP图像生成： 1. 对敏感图像的低分辨率小波系数进行差分隐私微调训练自回归光谱图像标记模型。 2. 使用公共预训练的超分辨率模型执行高分辨率上采样。 通过在第一阶段将隐私预算限制在图像的整体结构上，并利用DP的事后处理属性来细化细节，我们实现了隐私和效用之间的良好权衡。实验表明，在MS-COCO和MM-CelebA-HQ数据集上的实验结果证明了我们的方法生成的图像具有更好的质量和风格捕捉能力，相比其他领先的DP图像框架而言有显著改善。
URL
https://arxiv.org/abs/2602.23262
