Abstract
High-quality data is essential for modern machine learning, yet many valuable corpora are sensitive and cannot be freely shared. Synthetic data offers a practical substitute for downstream development, and large language models (LLMs) have emerged as powerful engines for generating it. However, existing private text generation methods are severely inefficient: they are data-intensive, computationally slow, and often require large private corpora or batch sizes to achieve usable quality. We introduce EPSVec, a differentially-private lightweight alternative that steers LLM generation using *dataset vectors*--directions in activation space that capture the distributional gap between private data and public priors. EPSVec extracts and sanitizes steering vectors just once and then performs standard decoding. This decouples the privacy budget from generation, enabling arbitrarily many synthetic samples without additional privacy cost and yielding strong fidelity even in low-data regimes. Furthermore, we enhance our method by utilizing pretrained (base) models and introducing fixed-shot prompting to boost generation diversity and fidelity. Our experiments demonstrate that EPSVec outperforms existing baselines in distributional alignment and downstream utility, particularly in low-data regimes, while significantly reducing computational overhead.
Abstract (translated)
高质量的数据对于现代机器学习至关重要，然而许多有价值的语料库包含敏感信息，无法自由共享。合成数据为下游开发提供了一个实用的替代方案，并且大型语言模型（LLM）已成为生成这类数据的强大引擎。然而，现有的私有文本生成方法效率极低：它们需要大量数据、计算缓慢，并且通常还需要庞大的私人语料库或批量大小才能达到可用的质量标准。我们引入了EPSVec，这是一种差分隐私的轻量级替代方案，通过使用“数据集向量”来引导LLM生成。“数据集向量”是在激活空间中捕捉私有数据和公共先验之间分布差距的方向。EPSVec只需一次提取并清理这些导向向量，并随后执行标准解码操作。这将隐私预算与生成过程分离，使得可以无限次地生成合成样本而无需额外的隐私成本，并且即使在低数据环境下也能保持强大的忠实度。此外，我们通过利用预训练模型和引入固定的提示技术来增强我们的方法，以提高生成多样性和忠实度。 实验结果表明，在分布对齐和下游应用性能方面，EPSVec优于现有的基准方法，特别是在低数据环境下的表现更为突出，并且显著减少了计算开销。
URL
https://arxiv.org/abs/2602.21218
