Abstract
Large language models (LLMs) are widely adopted to generate synthetic datasets for various natural language processing (NLP) tasks, such as text classification and summarization. However, accurately measuring the diversity of these synthetic datasets-an aspect crucial for robust model performance-remains a significant challenge. In this paper, we introduce DCScore, a novel method for measuring synthetic dataset diversity from a classification perspective. Specifically, DCScore formulates diversity evaluation as a sample classification task, leveraging mutual relationships among samples. We further provide theoretical verification of the diversity-related axioms satisfied by DCScore, highlighting its role as a principled diversity evaluation method. Experimental results on synthetic datasets reveal that DCScore enjoys a stronger correlation with multiple diversity pseudo-truths of evaluated datasets, underscoring its effectiveness. Moreover, both empirical and theoretical evidence demonstrate that DCScore substantially reduces computational costs compared to existing approaches. Code is available at: this https URL.
Abstract (translated)
大型语言模型（LLMs）被广泛应用于生成各种自然语言处理（NLP）任务的合成数据集，例如文本分类和摘要。然而，准确衡量这些合成数据集的多样性——这对于实现稳健的模型性能至关重要——仍然是一个重大挑战。在本文中，我们介绍了DCScore，这是一种从分类角度测量合成数据集多样性的新方法。具体来说，DCScore将多样性评估表述为一个样本分类任务，并利用了样本之间的相互关系。此外，我们还提供了关于DCScore所满足的多样性相关公理的理论验证，强调其作为原则性多样性评价方法的角色。在合成数据集上的实验结果表明，DCScore与被评估的数据集中多个多样性的伪真实情况之间存在更强的相关性，这凸显了它的有效性。而且，无论是经验还是理论证据都证明，相较于现有方法，DCScore能够显著降低计算成本。代码可在以下链接获取：[此URL]（请将“this https URL”替换为实际提供的GitHub或其它版本控制系统的链接）。
URL
https://arxiv.org/abs/2502.08512
