Abstract
Automatic text classification (ATC) has experienced remarkable advancements in the past decade, best exemplified by recent small and large language models (SLMs and LLMs), leveraged by Transformer architectures. Despite recent effectiveness improvements, a comprehensive cost-benefit analysis investigating whether the effectiveness gains of these recent approaches compensate their much higher costs when compared to more traditional text classification approaches such as SVMs and Logistic Regression is still missing in the literature. In this context, this work's main contributions are twofold: (i) we provide a scientifically sound comparative analysis of the cost-benefit of twelve traditional and recent ATC solutions including five open LLMs, and (ii) a large benchmark comprising {22 datasets}, including sentiment analysis and topic classification, with their (train-validation-test) partitions based on folded cross-validation procedures, along with documentation, and code. The release of code, data, and documentation enables the community to replicate experiments and advance the field in a more scientifically sound manner. Our comparative experimental results indicate that LLMs outperform traditional approaches (up to 26%-7.1% on average) and SLMs (up to 4.9%-1.9% on average) in terms of effectiveness. However, LLMs incur significantly higher computational costs due to fine-tuning, being, on average 590x and 8.5x slower than traditional methods and SLMs, respectively. Results suggests the following recommendations: (1) LLMs for applications that require the best possible effectiveness and can afford the costs; (2) traditional methods such as Logistic Regression and SVM for resource-limited applications or those that cannot afford the cost of tuning large LLMs; and (3) SLMs like Roberta for near-optimal effectiveness-efficiency trade-off.
Abstract (translated)
自动文本分类（ATC）在过去十年中取得了显著的进展，这一进步最典型的例子是近期的小型和大型语言模型（SLMs 和 LLMs），这些模型采用了Transformer架构。尽管最近的有效性有了很大的提高，但文献中仍然缺乏一种全面的成本效益分析来研究这些新方法的效果提升是否能够补偿其相比于传统的文本分类方法如支持向量机(SVM)和逻辑回归等所增加的高昂成本。 在这个背景下，这项工作的主要贡献包括两个方面：(i) 我们提供了一种科学合理的比较分析，评估了十二种传统和近期ATC解决方案的成本效益，其中包括五种开源大型语言模型；(ii) 一个大规模基准测试集合，包含22个数据集（如情感分析和主题分类），这些数据集基于折叠交叉验证过程划分的训练、验证和测试部分，并附有文档和代码。代码、数据及文档的发布使得社区能够复制实验并以更科学的方式推进领域研究。 我们的比较实验结果表明，在有效性方面，大型语言模型(LLMs)优于传统方法（平均提升26%-7.1%）和小型语言模型(SLMs)（平均提升4.9%-1.9%）。然而，LLMs由于需要进行微调而产生显著更高的计算成本，与传统方法相比平均慢590倍，比SLMs慢8.5倍。实验结果表明以下建议：(1) 对于那些要求最佳性能且能承担相应费用的应用程序，推荐使用大型语言模型；(2) 资源有限或无法承受微调大LLM成本的场景，则适合采用传统方法如逻辑回归和SVM；(3) 需要接近最优效率-效果平衡的应用场景可以考虑选择Roberta等小型语言模型。
URL
https://arxiv.org/abs/2504.01930
