Abstract
With the emergence of ChatGPT, Transformer models have significantly advanced text classification and related tasks. Decoder-only models such as Llama exhibit strong performance and flexibility, yet they suffer from inefficiency on inference due to token-by-token generation, and their effectiveness in text classification tasks heavily depends on prompt quality. Moreover, their substantial GPU resource requirements often limit widespread adoption. Thus, the question of whether smaller language models are capable of effectively handling text classification tasks emerges as a topic of significant interest. However, the selection of appropriate models and methodologies remains largely underexplored. In this paper, we conduct a comprehensive evaluation of prompt engineering and supervised fine-tuning methods for transformer-based text classification. Specifically, we focus on practical industrial scenarios, including email classification, legal document categorization, and the classification of extremely long academic texts. We examine the strengths and limitations of smaller models, with particular attention to both their performance and their efficiency in Video Random-Access Memory (VRAM) utilization, thereby providing valuable insights for the local deployment and application of compact models in industrial settings.
Abstract (translated)
随着ChatGPT的出现，Transformer模型在文本分类及相关任务上取得了显著的进步。例如Llama这样的解码器自回归模型展现出强大的性能和灵活性，但由于逐令牌生成过程中的效率低下，在推理阶段表现不佳，并且它们在文本分类任务上的效果严重依赖于提示的质量。此外，这些模型对GPU资源的巨大需求往往限制了其广泛应用。因此，关于较小规模的语言模型是否能够有效处理文本分类任务的问题引起了广泛关注。然而，选择合适的模型和方法仍然是一个未被充分探索的领域。 本文中，我们全面评估了基于Transformer的文本分类中的提示工程和监督微调方法。具体来说，我们的研究集中在实际工业场景上，包括电子邮件分类、法律文档归类以及对极其长篇学术文章进行分类的任务。我们将重点关注小型模型的优势与局限性，并特别关注它们在性能及视频随机访问存储器（VRAM）利用率方面的表现，从而为紧凑型模型在工业环境中的本地部署和应用提供宝贵的见解。
URL
https://arxiv.org/abs/2505.16078
