Abstract
The tension between data privacy and model utility has become the defining bottleneck for the practical deployment of large language models (LLMs) trained on sensitive corpora including healthcare. Differentially private stochastic gradient descent (DP-SGD) guarantees formal privacy, yet it does so at a pronounced cost: gradients are forcibly clipped and perturbed with noise, degrading sample efficiency and final accuracy. Numerous variants have been proposed to soften this trade-off, but they all share a handicap: their control knobs are hard-coded, global, and oblivious to the evolving optimization landscape. Consequently, practitioners are forced either to over-spend privacy budget in pursuit of utility, or to accept mediocre models in order to stay within privacy constraints. We present RLDP, the first framework to cast DP optimization itself as a closed-loop control problem amenable to modern deep reinforcement learning (RL). RLDP continuously senses rich statistics of the learning dynamics and acts by selecting fine-grained per parameter gradient-clipping thresholds as well as the magnitude of injected Gaussian noise. A soft actor-critic (SAC) hyper-policy is trained online during language model fine-tuning; it learns, from scratch, how to allocate the privacy budget where it matters and when it matters. Across more than 1,600 ablation experiments on GPT2-small, Llama-1B, Llama-3B, and Mistral-7B, RLDP delivers perplexity reductions of 1.3-30.5% (mean 5.4%) and an average 5.6% downstream utility gain. RLDP reaches each baseline's final utility after only 13-43% of the gradient-update budget (mean speed-up 71%), all while honoring the same ($\epsilon$, $\delta$)-DP contract and exhibiting equal or lower susceptibility to membership-inference and canary-extraction attacks.
Abstract (translated)
在敏感语料库（包括医疗保健）上训练的大规模语言模型（LLMs）的实际部署中，数据隐私与模型效用之间的紧张关系已成为主要瓶颈。差分隐私随机梯度下降（DP-SGD）提供了正式的隐私保证，但代价是显著的：它强制将梯度裁剪并添加噪声，从而降低样本效率和最终准确性。尽管提出了许多变体以缓解这种权衡，但它们都存在一个共同的问题：其控制参数是硬编码、全局性的，并且无视不断变化的优化景观。因此，实践者要么被迫过度使用隐私预算来追求效用，要么接受中等水平的质量模型以符合隐私限制。 我们介绍了RLDP框架，这是第一个将差分隐私优化本身作为现代深度强化学习（RL）可处理的闭环控制问题进行建模的方法。RLDP通过选择细化的逐参数梯度裁剪阈值以及注入高斯噪声的数量来连续感知学习动态的丰富统计信息并采取行动。在语言模型微调期间在线训练软演员评论家（SAC）超策略；它从零开始学习如何在其重要的领域和重要时刻分配隐私预算。 在GPT2-small、Llama-1B、Llama-3B 和 Mistral-7B 上进行的超过 1,600 次消融实验中，RLDP 实现了困惑度降低了 1.3% 到 30.5%（平均降低 5.4%），并且下游效用平均提高了 5.6%。RLDP 在仅使用每个基线最终效用所需的梯度更新预算的13%-43%时就达到了每个基准的最终效用，同时遵守相同的($\epsilon$, $\delta$)-差分隐私合同，并且表现出相同或更低的成员推断和金丝雀提取攻击易受性。
URL
https://arxiv.org/abs/2507.22565
