Abstract
Unlocking the potential of Large Language Models (LLMs) in data classification represents a promising frontier in natural language processing. In this work, we evaluate the performance of different LLMs in comparison with state-of-the-art deep-learning and machine-learning models, in two different classification scenarios: i) the classification of employees' working locations based on job reviews posted online (multiclass classification), and 2) the classification of news articles as fake or not (binary classification). Our analysis encompasses a diverse range of language models differentiating in size, quantization, and architecture. We explore the impact of alternative prompting techniques and evaluate the models based on the weighted F1-score. Also, we examine the trade-off between performance (F1-score) and time (inference response time) for each language model to provide a more nuanced understanding of each model's practical applicability. Our work reveals significant variations in model responses based on the prompting strategies. We find that LLMs, particularly Llama3 and GPT-4, can outperform traditional methods in complex classification tasks, such as multiclass classification, though at the cost of longer inference times. In contrast, simpler ML models offer better performance-to-time trade-offs in simpler binary classification tasks.
Abstract (translated)
解锁大型语言模型（LLMs）在数据分类中的潜力代表了自然语言处理领域的一个有前景的前沿方向。在这项工作中，我们评估了不同LLM与最先进的深度学习和机器学习模型相比，在两种不同的分类场景下的性能：i) 根据在线发布的职位评价来分类员工的工作地点（多类分类），以及 2) 将新闻文章分类为假新闻或非假新闻（二元分类）。我们的分析涵盖了各种在大小、量化和架构上有所区别的语言模型。我们探讨了不同的提示技术的影响，并根据加权F1分数对这些模型进行评估。此外，我们还考察了性能（F1得分）与时间（推理响应时间）之间的权衡关系，以提供每种语言模型实际应用中更细致的理解。我们的研究揭示了基于不同提示策略的模型响应存在显著差异。我们发现，特别是Llama3和GPT-4这样的LLM在复杂的分类任务（如多类分类）上可以超越传统方法，尽管这会带来推理时间较长的问题。相反，在较为简单的二元分类任务中，简单的机器学习模型提供了更好的性能与时间的权衡关系。
URL
https://arxiv.org/abs/2501.08457
