Abstract
Recently, Large Language Models (LLMs) have demonstrated significant potential for data annotation, markedly reducing the labor costs associated with downstream applications. However, existing methods mostly adopt an aggressive strategy by prompting LLM to determine a single gold label for each unlabeled sample. Due to the inherent uncertainty within LLMs, they often produce incorrect labels for difficult samples, severely compromising the data quality for downstream applications. Motivated by ambiguity aversion in human behaviors, we propose a novel candidate annotation paradigm wherein large language models are encouraged to output all possible labels when incurring uncertainty. To ensure unique labels are provided for downstream tasks, we develop a teacher-student framework CanDist that distills candidate annotations with a Small Language Model (SLM). We further provide a rigorous justification demonstrating that distilling candidate annotations from the teacher LLM offers superior theoretical guarantees compared to directly using single annotations. Extensive experiments across six text classification tasks validate the effectiveness of our proposed method. The source code is available at this https URL.
Abstract (translated)
最近，大型语言模型（LLMs）在数据标注方面展示了巨大的潜力，显著降低了下游应用的劳动力成本。然而，现有方法大多采用激进策略，通过提示LLM为每个未标记样本确定单一的“黄金标签”。由于LLM固有的不确定性，它们经常会为困难样本生成错误的标签，严重损害了下游应用程序的数据质量。 受到人类行为中规避模糊性的启发，我们提出了一种新颖的候选标注范式，在遇到不确定情况时鼓励大型语言模型输出所有可能的标签。为了确保为下游任务提供唯一的标签，我们开发了一个教师-学生框架CanDist，利用小型语言模型（SLM）对候选标注进行蒸馏。进一步地，我们提供了严格的理论依据，证明从老师LLM中蒸馏出候选注解相比直接使用单一注解具有更好的理论保证。 在六种文本分类任务上进行的广泛实验验证了我们提出的方法的有效性。源代码可在以下链接获得：[此处插入具体URL]。
URL
https://arxiv.org/abs/2506.03857
