Abstract
As a fundamental task in machine learning, text classification plays a crucial role in many areas. With the rapid scaling of Large Language Models (LLMs), particularly through reinforcement learning (RL), there is a growing need for more capable discriminators. Consequently, advances in classification are becoming increasingly vital for enhancing the overall capabilities of LLMs. Traditional discriminative methods map text to labels but overlook LLMs' intrinsic generative strengths. Generative classification addresses this by prompting the model to directly output labels. However, existing studies still rely on simple SFT alone, seldom probing the interplay between training and inference prompts, and no work has systematically leveraged RL for generative text classifiers and unified SFT, RL, and inference-time prompting in one framework. We bridge this gap with GenCLS++, a framework that jointly optimizes SFT and RL while systematically exploring five high-level strategy dimensions-in-context learning variants, category definitions, explicit uncertainty labels, semantically irrelevant numeric labels, and perplexity-based decoding-during both training and inference. After an SFT "policy warm-up," we apply RL with a simple rule-based reward, yielding sizable extra gains. Across seven datasets, GenCLS++ achieves an average accuracy improvement of 3.46% relative to the naive SFT baseline; on public datasets, this improvement rises to 4.00%. Notably, unlike reasoning-intensive tasks that benefit from explicit thinking processes, we find that classification tasks perform better without such reasoning steps. These insights into the role of explicit reasoning provide valuable guidance for future LLM applications.
Abstract (translated)
在机器学习中，文本分类是一项基本任务，在许多领域都发挥着关键作用。随着大型语言模型（LLM）通过强化学习（RL）迅速扩展，对更强大的判别器的需求日益增长。因此，推进分类技术对于增强LLM的整体能力变得越来越重要。传统的判别方法将文本映射到标签，但忽视了LLM内在的生成优势。而生成式分类则通过促使模型直接输出标签来解决这一问题。然而，现有的研究大多依赖于简单的指令微调（SFT）方法，并且很少探讨训练和推理提示之间的相互作用，也没有任何工作系统地利用RL为生成式的文本分类器并统一SFT、RL以及推理时的提示技术在同一个框架内进行部署。 我们通过GenCLS++填补了这一空白。这是一个优化SFT和RL的同时，在训练与推理过程中系统性探索五个高层次策略维度——上下文学习变体、类别定义、显式不确定性标签、语义无关数字标签及基于困惑度解码的框架。在经过SFT“政策预热”后，我们采用了一个简单的规则基础奖励机制进行强化学习，并获得了显著的额外增益。在七个数据集上，GenCLS++相对于朴素的SFT基线模型平均准确率提高了3.46%；而在公共数据集上的改进则上升到了4.00%。 值得注意的是，不同于需要明确思考过程的任务（如推理任务），我们发现分类任务在没有此类推理步骤的情况下表现得更好。这些关于显式推理作用的见解为未来LLM的应用提供了宝贵的指导原则。
URL
https://arxiv.org/abs/2504.19898
