Abstract
Differentially Private Stochastic Gradient Descent (DP-SGD) is a widely adopted technique for privacy-preserving deep learning. A critical challenge in DP-SGD is selecting the optimal clipping threshold C, which involves balancing the trade-off between clipping bias and noise magnitude, incurring substantial privacy and computing overhead during hyperparameter tuning. In this paper, we propose Dynamic Clipping DP-SGD (DC-SGD), a framework that leverages differentially private histograms to estimate gradient norm distributions and dynamically adjust the clipping threshold C. Our framework includes two novel mechanisms: DC-SGD-P and DC-SGD-E. DC-SGD-P adjusts the clipping threshold based on a percentile of gradient norms, while DC-SGD-E minimizes the expected squared error of gradients to optimize C. These dynamic adjustments significantly reduce the burden of hyperparameter tuning C. The extensive experiments on various deep learning tasks, including image classification and natural language processing, show that our proposed dynamic algorithms achieve up to 9 times acceleration on hyperparameter tuning than DP-SGD. And DC-SGD-E can achieve an accuracy improvement of 10.62% on CIFAR10 than DP-SGD under the same privacy budget of hyperparameter tuning. We conduct rigorous theoretical privacy and convergence analyses, showing that our methods seamlessly integrate with the Adam optimizer. Our results highlight the robust performance and efficiency of DC-SGD, offering a practical solution for differentially private deep learning with reduced computational overhead and enhanced privacy guarantees.
Abstract (translated)
差分隐私随机梯度下降（DP-SGD）是用于保护隐私的深度学习中广泛采用的技术。在DP-SGD中，选择最优剪裁阈值C是一个关键挑战，它需要在剪裁偏差和噪声大小之间找到平衡点，在超参数调整过程中这会产生大量的计算开销和隐私成本。本文提出了动态剪裁差分隐私随机梯度下降（DC-SGD），这是一种利用差分隐私直方图来估计梯度范数分布，并动态调整剪裁阈值C的框架。我们的框架包括两种新的机制：DC-SGD-P 和 DC-SGD-E。其中，DC-SGD-P 基于梯度范数的某个百分位数值来调整剪裁阈值，而DC-SGD-E 则通过最小化梯度平方误差的期望值来优化C值。这些动态调整显著减少了超参数调优过程中关于选择最优C值的工作量。 在各种深度学习任务上的广泛实验（包括图像分类和自然语言处理）表明，我们的动态算法比DP-SGD快9倍。特别是在给定相同的隐私预算的情况下，在CIFAR10数据集上，DC-SGD-E 达到了比 DP-SGD 高出10.62%的准确性。 我们进行了严格的理论分析，证明了我们的方法可以与Adam优化器无缝集成，并展示了 DC-SGD 在隐私保护和收敛性方面的稳健性能及高效性。这些结果表明，在减少计算开销的同时增强隐私保证方面，DC-SGD 提供了一种实用解决方案，适用于差分隐私深度学习的广泛应用场景。
URL
https://arxiv.org/abs/2503.22988
