Abstract
Using LLM-generated labels to fine-tune smaller encoder-only models for text classification has gained popularity in various settings. While this approach may be justified in simple and low-stakes applications, we conduct empirical analysis to demonstrate how the perennial curse of training on synthetic data manifests itself in this specific setup. Compared to models trained on gold labels, we observe not only the expected performance degradation in accuracy and F1 score, but also increased instability across training runs and premature performance plateaus. These findings cast doubts on the reliability of such approaches in real-world applications. We contextualize the observed phenomena through the lens of error propagation and offer several practical mitigation strategies, including entropy-based filtering and ensemble techniques. Although these heuristics offer partial relief, they do not fully resolve the inherent risks of propagating non-random errors from LLM annotations to smaller classifiers, underscoring the need for caution when applying this workflow in high-stakes text classification tasks.
Abstract (translated)
使用大型语言模型（LLM）生成的标签来微调较小的编码器-only模型进行文本分类，在各种场景中越来越受欢迎。尽管这种方法在简单和低风险的应用程序中可能是合理的，但我们通过实证分析展示了在这种特定设置下训练时使用合成数据所带来的长期困扰如何体现出来。与基于黄金标准标签训练的模型相比，我们不仅观察到了预期中的准确率和F1分数下降，还发现了跨训练运行的不稳定性增加以及性能过早达到饱和的现象。这些发现对这种方法在现实世界应用中的可靠性提出了质疑。 我们通过错误传播的角度来解释所观察到的现象，并提出了一些实用的缓解策略，包括基于熵的过滤方法和集成技术。尽管这些经验法则提供了一定程度上的缓解效果，但它们并不能完全解决将非随机错误从LLM注释传递给较小分类器的根本风险问题，这强调了在高风险文本分类任务中应用这一工作流程时需谨慎行事的重要性。
URL
https://arxiv.org/abs/2504.15432
