Abstract
Large Language Models (LLMs) like GPT-4o can help automate text classification tasks at low cost and scale. However, there are major concerns about the validity and reliability of LLM outputs. By contrast, human coding is generally more reliable but expensive to procure at scale. In this study, we propose a hybrid solution to leverage the strengths of both. We combine human-coded data and synthetic LLM-produced data to fine-tune a classical machine learning classifier, distilling both into a smaller BERT model. We evaluate our method on a human-coded test set as a validity measure for LLM output quality. In three experiments, we systematically vary LLM-generated samples' size, variety, and consistency, informed by best practices in LLM tuning. Our findings indicate that augmenting datasets with synthetic samples improves classifier performance, with optimal results achieved at an 80% synthetic to 20% human-coded data ratio. Lower temperature settings of 0.3, corresponding to less variability in LLM generations, produced more stable improvements but also limited model learning from augmented samples. In contrast, higher temperature settings (0.7 and above) introduced greater variability in performance estimates and, at times, lower performance. Hence, LLMs may produce more uniform output that classifiers overfit to earlier or produce more diverse output that runs the risk of deteriorating model performance through information irrelevant to the prediction task. Filtering out inconsistent synthetic samples did not enhance performance. We conclude that integrating human and LLM-generated data to improve text classification models in assessment offers a scalable solution that leverages both the accuracy of human coding and the variety of LLM outputs.
Abstract (translated)
大型语言模型（LLMs）如GPT-4可以以低成本和大规模的方式自动化文本分类任务，但其输出的有效性和可靠性存在重大问题。相比之下，人工编码的数据通常更可靠，但在大规模获取时成本高昂。在本研究中，我们提出了一种混合解决方案，旨在结合两者的优势。我们将人工编码数据与LLM生成的合成数据结合起来，对经典的机器学习分类器进行微调，并将这些数据浓缩到一个较小的BERT模型中。我们在一个人工编码的测试集上评估了我们的方法的有效性，作为衡量LLM输出质量的标准。在三项实验中，我们系统地改变了LLM生成样本的数量、种类和一致性，参考了最佳的LLM调优实践。我们的发现表明，通过使用合成样本增强数据集可以提高分类器的性能，在80%合成样本与20%人工编码数据的比例下达到最优结果。 较低的温度设置（如0.3）对应于更少的生成变异性，并产生了更加稳定的改进效果，但也限制了模型从扩充的数据中学习。相比之下，较高的温度设置（0.7及以上）引入了更大的性能估计变动性，在某些情况下甚至导致性能下降。因此，LLM可能产生分类器会过度拟合的较为统一的输出，或者生成多样化的输出从而带来风险，即通过与预测任务无关的信息降低模型性能。 过滤不一致的合成样本并未提高性能表现。综上所述，结合人工和LLM生成的数据以改进评估中的文本分类模型提供了一种可扩展的解决方案，该方案利用了人工编码的准确性以及LLM输出的多样性。
URL
https://arxiv.org/abs/2501.09126
