Abstract
Efficient text classification is essential for handling the increasing volume of academic publications. This study explores the use of pre-trained language models (PLMs), including BERT, SciBERT, BioBERT, and BlueBERT, fine-tuned on the Web of Science (WoS-46985) dataset for scientific text classification. To enhance performance, we augment the dataset by executing seven targeted queries in the WoS database, retrieving 1,000 articles per category aligned with WoS-46985's main classes. PLMs predict labels for this unlabeled data, and a hard-voting strategy combines predictions for improved accuracy and confidence. Fine-tuning on the expanded dataset with dynamic learning rates and early stopping significantly boosts classification accuracy, especially in specialized domains. Domain-specific models like SciBERT and BioBERT consistently outperform general-purpose models such as BERT. These findings underscore the efficacy of dataset augmentation, inference-driven label prediction, hard-voting, and fine-tuning techniques in creating robust and scalable solutions for automated academic text classification.
Abstract (translated)
高效的文本分类对于处理不断增加的学术出版物数量至关重要。本研究探讨了预训练语言模型（PLM），包括BERT、SciBERT、BioBERT和BlueBERT，在Web of Science (WoS-46985) 数据集上进行微调，以实现科学文献分类的应用。为了增强性能，我们通过执行七个针对特定领域的查询来扩充该数据集，在WoS数据库中为每个类别检索1,000篇与WoS-46985主要类别对齐的文章。PLM模型对这些未标记的数据进行标签预测，并采用硬投票策略结合多个预测结果以提高准确性和置信度。在扩展后的数据集上使用动态学习率和提前停止法进行微调，显著提升了分类准确性，特别是在专业领域内表现尤为突出。特定领域的模型如SciBERT和BioBERT始终优于通用模型如BERT的性能。这些发现强调了数据扩充、基于推理的标签预测、硬投票以及微调技术在构建稳健且可扩展的自动学术文本分类解决方案中的有效性。
URL
https://arxiv.org/abs/2504.19021
