Abstract
Large language models (LLMs) such as GPT-4o and Claude Sonnet 4.5 have demonstrated strong capabilities in open-ended reasoning and generative language tasks, leading to their widespread adoption across a broad range of NLP applications. However, for structured text classification problems with fixed label spaces, model selection is often driven by predictive performance alone, overlooking operational constraints encountered in production systems. In this work, we present a systematic comparison of two contrasting paradigms for text classification: zero- and few-shot prompt-based large language models, and fully fine-tuned encoder-only architectures. We evaluate these approaches across four canonical benchmarks (IMDB, SST-2, AG News, and DBPedia), measuring predictive quality (macro F1), inference latency, and monetary cost. We frame model evaluation as a multi-objective decision problem and analyze trade-offs using Pareto frontier projections and a parameterized utility function reflecting different deployment regimes. Our results show that fine-tuned encoder-based models from the BERT family achieve competitive, and often superior, classification performance while operating at one to two orders of magnitude lower cost and latency compared to zero- and few-shot LLM prompting. Overall, our findings suggest that indiscriminate use of large language models for standard text classification workloads can lead to suboptimal system-level outcomes. Instead, fine-tuned encoders emerge as robust and efficient components for structured NLP pipelines, while LLMs are better positioned as complementary elements within hybrid architectures. We release all code, datasets, and evaluation protocols to support reproducibility and cost-aware NLP system design.
Abstract (translated)
大型语言模型（LLMs）如GPT-4o和Claude Sonnet 4.5在开放性推理和生成语言任务中展现了强大的能力，因此这些模型被广泛应用于各种自然语言处理（NLP）应用。然而，在结构化的文本分类问题中，这些问题具有固定的标签空间时，模型的选择往往只关注预测性能，而忽略了生产系统中的操作限制。在这项工作中，我们提出了两种截然不同的文本分类方法的系统比较：零样本和少量样本文本提示的大语言模型与完全微调过的编码器架构。我们在四个标准基准（IMDB、SST-2、AG News 和 DBPedia）上评估了这些方法，并测量了预测质量（宏F1分数）、推理延迟以及经济成本。我们将模型评估框架化为一个多目标决策问题，并通过帕累托前沿投影和反映不同部署方案的参数化效用函数来分析权衡。我们的结果表明，基于BERT家族的完全微调编码器模型在分类性能上具有竞争力甚至更优，同时其运行成本和延迟仅为零样本或少量样本文本提示的大语言模型的一到两个数量级。 总体而言，我们的研究发现认为，在标准文本分类工作负载中无差别地使用大型语言模型会导致次优的系统级结果。相反，微调过的编码器作为结构化NLP管道中的稳健且高效的组件而出现，而LLM则更适合在混合架构中作为补充元素。我们发布了所有代码、数据集和评估协议以支持可重复性以及成本意识型NLP系统的构建。
URL
https://arxiv.org/abs/2602.06370
