Abstract
The rapid adoption of LLMs has overshadowed the potential advantages of traditional BERT-like models in text classification. This study challenges the prevailing "LLM-centric" trend by systematically comparing three category methods, i.e., BERT-like models fine-tuning, LLM internal state utilization, and zero-shot inference across six high-difficulty datasets. Our findings reveal that BERT-like models often outperform LLMs. We further categorize datasets into three types, perform PCA and probing experiments, and identify task-specific model strengths: BERT-like models excel in pattern-driven tasks, while LLMs dominate those requiring deep semantics or world knowledge. Based on this, we propose TaMAS, a fine-grained task selection strategy, advocating for a nuanced, task-driven approach over a one-size-fits-all reliance on LLMs.
Abstract (translated)
大型语言模型（LLMs）的迅速采用已经掩盖了传统BERT类模型在文本分类中的潜在优势。本研究通过系统地比较三种类别方法——即对BERT类模型进行微调、利用LLM内部状态以及零样本推理，在六个高难度数据集上挑战现有的“LLM中心论”趋势。我们的发现表明，传统的BERT类模型往往优于大型语言模型（LLMs）。我们进一步将数据集分类为三类，并进行了PCA和探测实验，以识别任务特定的模型优势：BERT类模型在基于模式的任务中表现出色，而大型语言模型则在需要深度语义或世界知识的任务中占据主导地位。基于此，我们提出了TaMAS，一种细粒度的任务选择策略，提倡采取精细、任务驱动的方法而非一刀切地依赖LLMs。
URL
https://arxiv.org/abs/2505.18215
