Abstract
We present PRINCIPLE-BASED PROMPTING, a simple but effective multi-agent prompting strategy for text classification. It first asks multiple LLM agents to independently generate candidate principles based on analysis of demonstration samples with or without labels, consolidates them into final principles via a finalizer agent, and then sends them to a classifier agent to perform downstream classification tasks. Extensive experiments on binary and multi-class classification datasets with different sizes of LLMs show that our approach not only achieves substantial performance gains (1.55% - 19.37%) over zero-shot prompting on macro-F1 score but also outperforms other strong baselines (CoT and stepback prompting). Principles generated by our approach help LLMs perform better on classification tasks than human crafted principles on two private datasets. Our multi-agent PRINCIPLE-BASED PROMPTING approach also shows on-par or better performance compared to demonstration-based few-shot prompting approaches, yet with substantially lower inference costs. Ablation studies show that label information and the multi-agent cooperative LLM framework play an important role in generating high-quality principles to facilitate downstream classification tasks.
Abstract (translated)
我们提出了一种基于原则的提示策略（PRINCIPLE-BASED PROMPTING），这是一种简单但有效的多代理文本分类提示方法。首先，该方法让多个大型语言模型（LLM）代理独立生成候选原则，这些候选原则是通过对带有或不带标签的示范样本进行分析得出的。随后，通过一个最终化器代理将这些候选原则整合成最终的原则，并将其发送给分类器代理以执行下游分类任务。 在不同大小的语言模型上的二元和多类分类数据集上进行了广泛的实验表明，我们的方法不仅在宏平均F1得分上比零样本提示获得了显著的性能提升（1.55% - 19.37%），而且也超过了其他强大的基准（如CoT 和 stepback 提示）。由我们方法生成的原则帮助语言模型在两个私有数据集上的分类任务中优于人工创建的原则。此外，我们的多代理 PRINCIPLE-BASED PROMPTING 方法显示出了与基于示范的少量样本提示方法相当或更好的性能表现，但推理成本却大大降低。 消融研究还表明，标签信息和多代理协同语言模型框架在生成高质量原则以促进下游分类任务方面发挥了重要作用。
URL
https://arxiv.org/abs/2502.07165
