Abstract
Large language models are increasingly used for patient-facing medical assistance and clinical decision support, but adapting them to clinical dialogue often requires supervision derived from doctor-patient conversations that may contain sensitive information. Conventional supervised fine-tuning and reinforcement learning from human feedback (RLHF) can amplify memorization risks, enabling empirical membership inference and extraction of rare training-set content. We present PrivMedChat, an end-to-end framework for differentially private RLHF (DP-RLHF) for medical dialogue. Our design enforces differential privacy at every training stage that directly accesses dialogue-derived supervision: (i) Differential Private Stochastic Gradient Descent (DP-SGD) for medical SFT and (ii) DP-SGD for reward model learning from preference pairs. To limit additional privacy expenditure during alignment, we apply DP-SGD to the PPO actor and critic when operating on dialogue-derived prompts, while the reward model remains fixed after DP training. We also introduce an annotation-free preference construction strategy that pairs physician responses with filtered non-expert generations to produce scalable preference data without clinician labeling. Experiments on medical dialogue benchmarks show that PrivMedChat at $\varepsilon=7$ achieves the highest ROUGE-L of 0.156 among all DP models, reduces clinical hallucinations to 1.4% and harmful advice to 0.4%, and obtains the highest overall score of 2.86 in a 3-model LLM-jury evaluation, while producing membership-inference signals that are near chance (AUC 0.510-0.555). We open-source our code at this https URL.
Abstract (translated)
大型语言模型在患者医疗服务和临床决策支持中的应用日益广泛，但要将这些模型适应于医疗对话通常需要基于医生与患者交流的监督数据，而这些数据可能包含敏感信息。传统的有监督微调（Supervised Fine-Tuning, SFT）和从人类反馈中进行强化学习（Reinforcement Learning from Human Feedback, RLHF）可能会放大记忆风险，从而使得经验性的成员推断成为可能，并且可以提取罕见的训练集内容。我们提出了PrivMedChat，这是一个用于医疗对话的差分隐私RLHF (DP-RLHF) 的端到端框架。我们的设计强制在每个直接访问从对话中获取监督数据的训练阶段实施差分隐私：(i) 对于医学SFT使用差分私有随机梯度下降（Differential Private Stochastic Gradient Descent, DP-SGD）；(ii) 在从偏好对学习奖励模型时也采用DP-SGD。为了在调整模型行为时限制额外的隐私开销，当处理由对话生成的提示时，我们只应用DP-SGD于PPO策略和价值函数（actor和critic），而经过差分隐私训练后的奖励模型则保持不变。此外，我们还提出了一种无需标注的偏好构建策略，该策略通过将医生的回答与过滤掉专家部分之后的非专家生成进行配对来创建可扩展且不需要临床医生标记的数据。 在医疗对话基准测试中的实验表明，在$\varepsilon=7$的情况下，PrivMedChat实现了所有差分隐私模型中最高的ROUGE-L值0.156，将临床假阳性率降低到1.4%，有害建议减少至0.4%，并在包含三个大型语言模型的评估中获得了最高总评分2.86。同时，在成员推断信号方面几乎接近随机（AUC 0.510-0.555）。我们的代码可以在这里访问：[此https URL]。 这种框架不仅保护了训练数据中的隐私信息，还在生成医疗服务对话时显著减少了潜在的临床风险和有害建议的产生。
URL
https://arxiv.org/abs/2603.03054
