Abstract
The exponential increase in scientific literature and online information necessitates efficient methods for extracting knowledge from textual data. Natural language processing (NLP) plays a crucial role in addressing this challenge, particularly in text classification tasks. While large language models (LLMs) have achieved remarkable success in NLP, their accuracy can suffer in domain-specific contexts due to specialized vocabulary, unique grammatical structures, and imbalanced data distributions. In this systematic literature review (SLR), we investigate the utilization of pre-trained language models (PLMs) for domain-specific text classification. We systematically review 41 articles published between 2018 and January 2024, adhering to the PRISMA statement (preferred reporting items for systematic reviews and meta-analyses). This review methodology involved rigorous inclusion criteria and a multi-step selection process employing AI-powered tools. We delve into the evolution of text classification techniques and differentiate between traditional and modern approaches. We emphasize transformer-based models and explore the challenges and considerations associated with using LLMs for domain-specific text classification. Furthermore, we categorize existing research based on various PLMs and propose a taxonomy of techniques used in the field. To validate our findings, we conducted a comparative experiment involving BERT, SciBERT, and BioBERT in biomedical sentence classification. Finally, we present a comparative study on the performance of LLMs in text classification tasks across different domains. In addition, we examine recent advancements in PLMs for domain-specific text classification and offer insights into future directions and limitations in this rapidly evolving domain.
Abstract (translated)
科学文献和在线信息的指数增长要求高效的方法从文本数据中提取知识。自然语言处理（NLP）在应对这一挑战方面发挥着关键作用，特别是在文本分类任务中。尽管大型语言模型（LLMs）在NLP领域取得了显著的成功，但在特定领域的背景下，由于专业词汇、独特的语法结构以及不平衡的数据分布等原因，它们的准确性可能会受到影响。在这次系统性文献综述（SLR）中，我们研究了预训练语言模型（PLMs）在特定领域文本分类中的应用。我们在2018年至2024年1月间共审查了41篇论文，并遵循PRISMA声明（系统评价和元分析的首选报告项目）。该综述方法采用了严格的纳入标准以及使用AI工具进行多步骤选择的过程。我们深入探讨了文本分类技术的发展历程，区分了传统方法与现代方法的区别。我们强调基于转换器的模型，并研究了在特定领域内应用LLMs时面临的挑战和考虑因素。此外，我们依据不同的PLMs对现有研究进行了分类，并提出了一种该领域的技术分类体系。为了验证我们的发现，我们在生物医学句子分类任务中进行了BERT、SciBERT和BioBERT之间的比较实验。最后，我们提供了不同领域中大型语言模型在文本分类任务性能上的对比分析。此外，我们还考察了PLMs在特定领域文本分类中的最新进展，并为该快速发展的领域的未来方向及限制性因素提供见解。
URL
https://arxiv.org/abs/2510.17892
