Abstract
Machine learning models for text classification are trained to predict a class for a given text. To do this, training and validation samples must be prepared: a set of texts is collected, and each text is assigned a class. These classes are usually assigned by human annotators with different expertise levels, depending on the specific classification task. Collecting such samples from scratch is labor-intensive because it requires finding specialists and compensating them for their work; moreover, the number of available specialists is limited, and their productivity is constrained by human factors. While it may not be too resource-intensive to collect samples once, the ongoing need to retrain models (especially in incremental learning pipelines) to address data drift (also called model drift) makes the data collection process crucial and costly over the model's entire lifecycle. This paper proposes several approaches to replace human annotators with Large Language Models (LLMs) to test classifier predictions for correctness, helping ensure model quality and support high-quality incremental learning.
Abstract (translated)
用于文本分类的机器学习模型是被训练来为给定的文本预测一个类别。为了实现这一点，必须准备训练和验证样本：收集一组文本，并且每个文本都由人工标注员分配一个类别。这些类通常是根据特定的分类任务的不同要求，由具有不同专业知识水平的人工标注员进行标注。从头开始收集这样的样本是非常费力的工作，因为它需要找到专家并支付他们的费用；此外，可用的专家数量有限，而且他们的工作效率受到人类因素的影响而受限。 虽然一次性收集一些样本可能不会消耗太多资源，但在整个模型生命周期中持续地为了应对数据漂移（也称为模型漂移）重新训练模型（特别是在增量学习管道中），使数据收集过程变得至关重要且成本高昂。本文提出了一些方法来用大型语言模型（LLMs）替代人工标注员以测试分类器预测的准确性，这有助于确保模型的质量，并支持高质量的增量学习。
URL
https://arxiv.org/abs/2505.18688
