Abstract
Counterfactual examples are frequently used for model development and evaluation in many natural language processing (NLP) tasks. Although methods for automated counterfactual generation have been explored, such methods depend on models such as pre-trained language models that are then fine-tuned on auxiliary, often task-specific datasets. Collecting and annotating such datasets for counterfactual generation is labor intensive and therefore, infeasible in practice. Therefore, in this work, we focus on a novel problem setting: \textit{zero-shot counterfactual generation}. To this end, we propose a structured way to utilize large language models (LLMs) as general purpose counterfactual example generators. We hypothesize that the instruction-following and textual understanding capabilities of recent LLMs can be effectively leveraged for generating high quality counterfactuals in a zero-shot manner, without requiring any training or fine-tuning. Through comprehensive experiments on various downstream tasks in natural language processing (NLP), we demonstrate the efficacy of LLMs as zero-shot counterfactual generators in evaluating and explaining black-box NLP models.
Abstract (translated)
反事实例子在许多自然语言处理（NLP）任务中常被用于模型开发和评估。尽管已经探索了自动反事实生成的方法，但这些方法依赖于诸如预训练语言模型这样的模型，然后在辅助、通常针对特定任务的辅助数据集上进行微调。因此，收集和标注这样的数据集对于反事实生成来说是不够的，在实践中是不可行的。因此，在本文中，我们关注一个新颖的问题场景：零击反事实生成。为此，我们提出了一个有结构的方法，将大型语言模型（LLMs）用作通用反事实例子生成器。我们假设，LLMs最近的指令跟随和文本理解能力可以有效地用于以零击的方式生成高质量的反事实，而无需进行训练或微调。通过对自然语言处理（NLP）中的各种下游任务的全面实验，我们证明了LLMs作为零击反事实生成器在评估和解释黑盒NLP模型方面的有效性。
URL
https://arxiv.org/abs/2405.04793
