Abstract
We introduce Co-DETECT (Collaborative Discovery of Edge cases in TExt ClassificaTion), a novel mixed-initiative annotation framework that integrates human expertise with automatic annotation guided by large language models (LLMs). Co-DETECT starts with an initial, sketch-level codebook and dataset provided by a domain expert, then leverages the LLM to annotate the data and identify edge cases that are not well described by the initial codebook. Specifically, Co-DETECT flags challenging examples, induces high-level, generalizable descriptions of edge cases, and assists user in incorporating edge case handling rules to improve the codebook. This iterative process enables more effective handling of nuanced phenomena through compact, generalizable annotation rules. Extensive user study, qualitative and quantitative analyses prove the effectiveness of Co-DETECT.
Abstract (translated)
我们介绍了Co-DETECT（文本分类中边缘情况的协作发现），这是一个新颖的混合主动注释框架，它结合了人类专业知识与大型语言模型（LLMs）指导下的自动标注。Co-DETECT始于领域专家提供的初步代码本和数据集，然后利用LLM对数据进行标注，并识别出那些初始代码本描述不充分的边缘案例。具体来说，Co-DETECT会标记具有挑战性的例子，引出高层次、可泛化的边缘案例描述，并帮助用户将处理边缘案例的规则纳入代码本中以改进其内容。这一迭代过程使得通过简洁且通用的注释规则更有效地处理细微现象成为可能。广泛的用户研究以及定性和定量分析证明了Co-DETECT的有效性。
URL
https://arxiv.org/abs/2507.05010
