Abstract
Etremely Weakly Supervised Text Classification (XWS-TC) refers to text classification based on minimal high-level human guidance, such as a few label-indicative seed words or classification instructions. There are two mainstream approaches for XWS-TC, however, never being rigorously compared: (1) training classifiers based on pseudo-labels generated by (softly) matching seed words (SEED) and (2) prompting (and calibrating) language models using classification instruction (and raw texts) to decode label words (PROMPT). This paper presents the first XWS-TC benchmark to compare the two approaches on fair grounds, where the datasets, supervisions, and hyperparameter choices are standardized across methods. Our benchmarking results suggest that (1) Both SEED and PROMPT approaches are competitive and there is no clear winner; (2) SEED is empirically more tolerant than PROMPT to human guidance (e.g., seed words, classification instructions, and label words) changes; (3) SEED is empirically more selective than PROMPT to the pre-trained language models; (4) Recent SEED and PROMPT methods have close connections and a clustering post-processing step based on raw in-domain texts is a strong performance booster to both. We hope this benchmark serves as a guideline in selecting XWS-TC methods in different scenarios and stimulate interest in developing guidance- and model-robust XWS-TC methods. We release the repo at this https URL.
Abstract (translated)
"Etremely Weakly Supervised Text Classification (XWS-TC)"指的是基于 minimal high-level human guidance，例如一些带有标签指示的种子词或分类指令的文字分类方法。然而，对于 XWS-TC 方法，从未进行严格的比较：(1) 基于(softly)匹配种子词生成伪标签的训练分类器(seed),(2) 使用分类指令(和原始文本)引导语言模型解码标签单词(PROMPT)。本文提出了第一个 XWS-TC 基准，旨在通过公正的理由比较两种方法，在该方法间进行数据集、监督器和超参数选择的统一。我们的基准测试结果显示，(1) 两个种子方法都是竞争力强的，没有明确的胜者；(2) 种子方法 empirical 上比 PROMPT 更加容忍人类指导的变化(例如种子词、分类指令和标签单词);(3) 种子方法 empirical 上比预训练语言模型更加选择性；(4) 最近的两个种子和 PROMPT 方法有密切的联系，基于原始领域文本的聚类后处理步骤是一个增强 both 方法性能的强大性能Booster。我们希望这个基准可以作为在不同情况下选择 XWS-TC 方法的指导方针，并刺激开发 guidance 和模型 robust XWS-TC 方法的兴趣。我们将在这个 https URL 上发布代码仓库。
URL
https://arxiv.org/abs/2305.12749
