Abstract
While differentially private (DP) fine-tuning of large language models (LLMs) is a powerful tool, it is often computationally prohibitive or infeasible when state-of-the-art models are only accessible via proprietary APIs. In such settings, generating DP synthetic data has emerged as a crucial alternative, offering the added benefits of arbitrary reuse across downstream tasks and transparent exploratory data analysis without the opaque constraints of a model's parameter space. Private Evolution (PE) is a promising API-based framework for this goal; however, its performance critically depends on initialization. When the private data distribution deviates substantially from the foundation model's pre-training priors--particularly in highly specialized domains--PE frequently struggles to align with the target data, resulting in degraded utility, poor convergence, and inefficient API usage. To address this initialization bottleneck, we propose Metadata Augmented Private Language Evolution (MAPLE). MAPLE leverages differentially private tabular metadata extraction and in-context learning to effectively ground the initial synthetic distribution in the target domain. Extensive experiments on challenging, domain-specific text generation tasks demonstrate that MAPLE achieves a significantly more favorable privacy-utility trade-off, converges faster, and drastically reduces API costs compared to previous PE methods.
Abstract (translated)
尽管对大语言模型进行差分隐私微调是一项强大工具，但当先进模型仅能通过专有API访问时，该技术通常面临计算成本过高或不可行的问题。在此类场景下，生成差分隐私合成数据已成为关键替代方案，其额外优势包括：可跨下游任务任意复用，且能在不触碰模型参数空间不透明限制的前提下实现透明的探索性数据分析。私有演化（PE）是实现此目标的前景广阔的API驱动框架；然而其性能严重依赖初始化。当私有数据分布与基础模型的预训练先验存在显著偏差时——尤其在高度专业化领域——PE常难以与目标数据对齐，导致效用下降、收敛困难及API使用效率低下。为突破此初始化瓶颈，我们提出元数据增强私有语言演化（MAPLE）。MAPLE利用差分隐私表格元数据提取与上下文学习，有效将初始合成分布锚定于目标领域。在具有挑战性的领域特定文本生成任务上的大量实验表明，相较于先前PE方法，MAPLE实现了更优的隐私-效用权衡、更快收敛速度，并大幅降低API成本。
URL
https://arxiv.org/abs/2603.19258
