Abstract
Differentially private (DP) synthetic data generation is a promising technique for utilizing private datasets that otherwise cannot be exposed for model training or other analytics. While much research literature has focused on generating private unstructured text and image data, in enterprise settings, structured data (e.g., tabular) is more common, often including natural language fields or components. Existing synthetic data evaluation techniques (e.g., FID) struggle to capture the structural properties and correlations of such datasets. In this work, we propose Struct-Bench, a framework and benchmark for evaluating synthetic datasets derived from structured datasets that contain natural language data. The Struct-Bench framework requires users to provide a representation of their dataset structure as a Context-Free Grammar (CFG). Our benchmark comprises 5 real-world and 2 synthetically generated datasets, each annotated with CFGs. We show that these datasets demonstrably present a great challenge even for state-of-the-art DP synthetic data generation methods. Struct-Bench also includes reference implementations of different metrics and a leaderboard, thereby providing researchers a standardized evaluation platform to benchmark and investigate privacy-preserving synthetic data generation methods. Further, we also present a case study showing how to use Struct-Bench to improve the synthetic data quality of Private Evolution (PE) on structured data. The benchmark and the leaderboard have been publicly made available at this https URL.
Abstract (translated)
差分隐私（DP）合成数据生成是一种有前景的技术，用于利用那些否则不能公开进行模型训练或其他分析的私人数据集。尽管许多研究文献主要关注于生成具有私密性的无结构文本和图像数据，但在企业环境中，结构化数据（例如表格数据）更为常见，并且通常包含自然语言字段或组件。现有的合成数据评估技术（如FID）难以捕捉此类数据集中的结构性属性和关联性。 在此工作中，我们提出了Struct-Bench——一个用于评估从含有自然语言数据的结构化数据集中生成的合成数据集的框架和基准。Struct-Bench框架要求用户提供其数据集结构的一种表示形式，即上下文无关语法（CFG）。我们的基准包括5个真实世界的数据集和2个合成生成的数据集，每个数据集都用CFG进行了注释。我们展示了这些数据集对最先进的DP合成数据生成方法构成了重大挑战。Struct-Bench还包括不同度量的参考实现以及排行榜，从而为研究人员提供了一个标准化的评估平台，以便对其隐私保护的合成数据生成方法进行基准测试和研究。 此外，我们也展示了一项案例研究，说明如何使用Struct-Bench来提高Private Evolution（PE）在结构化数据上的合成数据质量。该基准及排行榜现已公开发布，详情请访问此链接：[URL]。
URL
https://arxiv.org/abs/2509.10696
