Abstract
With the rapid advancement of synthetic dataset generation techniques, evaluating the quality of synthetic data has become a critical research focus. Robust evaluation not only drives innovations in data generation methods but also guides researchers in optimizing the utilization of these synthetic resources. However, current evaluation studies for synthetic datasets remain limited, lacking a universally accepted standard framework. To address this, this paper proposes a novel evaluation framework integrating generalized cross-validation experiments and domain transfer learning principles, enabling generalizable and comparable assessments of synthetic dataset quality. The framework involves training task-specific models (e.g., YOLOv5s) on both synthetic datasets and multiple real-world benchmarks (e.g., KITTI, BDD100K), forming a cross-performance matrix. Following normalization, a Generalized Cross-Validation (GCV) Matrix is constructed to quantify domain transferability. The framework introduces two key metrics. One measures the simulation quality by quantifying the similarity between synthetic data and real-world datasets, while another evaluates the transfer quality by assessing the diversity and coverage of synthetic data across various real-world scenarios. Experimental validation on Virtual KITTI demonstrates the effectiveness of our proposed framework and metrics in assessing synthetic data fidelity. This scalable and quantifiable evaluation solution overcomes traditional limitations, providing a principled approach to guide synthetic dataset optimization in artificial intelligence research.
Abstract (translated)
随着合成数据集生成技术的迅速发展，评估合成数据的质量已成为一个关键的研究焦点。稳健的评估不仅推动了数据生成方法的创新，还指导研究人员优化这些合成资源的应用。然而，目前对于合成数据集的评价研究仍然有限，并缺乏普遍接受的标准框架。为解决这一问题，本文提出了一种新的评估框架，该框架结合了广义交叉验证实验和领域转换学习原则，能够对合成数据的质量进行通用且可比较的评估。此框架涉及在合成数据集和多个真实世界基准（如KITTI、BDD100K）上训练特定任务的模型（例如YOLOv5s），从而形成一个跨性能矩阵。经过标准化处理后，构建广义交叉验证（GCV）矩阵来量化领域转换能力。该框架引入了两个关键指标：一个是通过量化合成数据与真实世界数据集之间的相似性来衡量模拟质量；另一个则是通过评估合成数据在各种现实场景中的多样性和覆盖范围来评定传输质量。基于虚拟KITTI的实验验证表明，我们提出的框架和度量方法能够有效地评估合成数据的真实度。这种可扩展且具量化性的评价解决方案克服了传统限制，为指导人工智能研究中合成数据集优化提供了一个原则性的方法。
URL
https://arxiv.org/abs/2509.11273
