Abstract
Language models are strong few-shot learners and achieve good overall accuracy in text classification tasks, masking the fact that their results suffer from great class accuracy imbalance. We believe that the pursuit of overall accuracy should not come from enriching the strong classes, but from raising up the weak ones. To address the imbalance, we propose a post-hoc nonlinear integer programming based debiasing method that ensembles weight correction and membership correction to enable flexible rectifications of class probabilities at both class and sample levels, enhancing the performance of LLMs directly from their outputs. Evaluations with Llama-2-13B on seven text classification benchmarks show that our approach achieves state-of-the-art overall accuracy gains with balanced class accuracies. The resulted probability correction scheme demonstrates that sample-level corrections are necessary to elevate weak classes. In addition, due to effectively correcting weak classes, our method also brings significant performance gains to Llama-2-70B, especially on a biomedical domain task, demonstrating its effectiveness across both small and large model variants.
Abstract (translated)
语言模型在少量样本学习任务中表现出色，并且在文本分类任务中的整体准确性很高，但这掩盖了一个事实：它们的结果存在严重的类别准确率不平衡。我们认为，不应通过增强强势类别来提高整体准确度，而应提升弱势类别的表现。为了解决这种不平衡问题，我们提出了一种基于后处理的非线性整数规划去偏方法，该方法结合了权重修正和成员身份修正，能够在分类级别和样本级别的灵活纠正类别概率，从而直接从语言模型输出中增强其性能。 通过使用Llama-2-13B在七个文本分类基准数据集上的评估显示，我们的方法实现了最先进的整体准确度提升，并且各类别的准确性更加均衡。所提出的概率修正方案表明，在提升弱势类别时，样本级别的修正至关重要。此外，由于有效纠正了弱势类别，我们提出的方法还为Llama-2-70B带来了显著的性能提升，尤其是在生物医学领域任务上，证明了其在小型和大型模型变体中的有效性。
URL
https://arxiv.org/abs/2503.05157
