Abstract
Eliciting explicit, step-by-step reasoning traces from large language models (LLMs) has emerged as a dominant paradigm for enhancing model capabilities. Although such reasoning strategies were originally designed for problems requiring explicit multi-step reasoning, they have increasingly been applied to a broad range of NLP tasks. This expansion implicitly assumes that deliberative reasoning uniformly benefits heterogeneous tasks. However, whether such reasoning mechanisms truly benefit classification tasks remains largely underexplored, especially considering their substantial token and time costs. To fill this gap, we introduce TextReasoningBench, a systematic benchmark designed to evaluate the effectiveness and efficiency of reasoning strategies for text classification with LLMs. We compare seven reasoning strategies, namely IO, CoT, SC-CoT, ToT, GoT, BoC, and long-CoT across ten LLMs on five text classification datasets. Beyond traditional metrics such as accuracy and macro-F1, we introduce two cost-aware evaluation metrics that quantify the performance gain per reasoning token and the efficiency of performance improvement relative to token cost growth. Experimental results reveal three notable findings: (1) Reasoning does not universally improve classification performance: while moderate strategies such as CoT and SC-CoT yield consistent but limited gains (typically +1% to +3% on big models), more complex methods (e.g., ToT and GoT) often fail to outperform simpler baselines and can even degrade performance, especially on small models; (2) Reasoning is often inefficient: many reasoning strategies increase token consumption by 10$\times$ to 100$\times$ (e.g., SC-CoT and ToT) while providing only marginal performance improvements.
Abstract (translated)
从大语言模型（LLMs）中提取显式的逐步推理痕迹，已成为提升模型能力的主流范式。尽管此类推理策略最初专为需要显式多步推理的问题设计，但现已日益广泛应用于各类自然语言处理任务。这种扩展隐含地假设审慎推理能普遍惠及异构任务。然而，此类推理机制是否真正有益于分类任务，尤其是考虑到其高昂的令牌与时间成本，仍 largely 未被充分探索。为填补这一空白，我们推出 TextReasoningBench——一个系统性基准测试，旨在评估 LLM 文本分类中推理策略的有效性与效率。我们在五个文本分类数据集上，对比了七种推理策略（IO、CoT、SC-CoT、ToT、GoT、BoC 及 long-CoT）在十种 LLM 上的表现。除准确率与宏 F1 等传统指标外，我们引入两项成本感知评估指标：量化每推理令牌的性能增益，以及性能提升效率相对于令牌成本增长的比率。实验结果显示三项关键发现：（1）推理并非普遍提升分类性能：适度策略（如 CoT 与 SC-CoT）在大模型上仅带来稳定但有限的增益（通常为 +1% 至 +3%），而复杂方法（如 ToT 与 GoT）常无法超越简单基线，甚至可能损害性能，在小模型上尤为明显；（2）推理往往效率低下：多数策略使令牌消耗增加 10 倍至 100 倍（如 SC-CoT 与 ToT），却仅换取微小的性能提升。
URL
https://arxiv.org/abs/2603.19558
