Abstract
Recent advances in natural language processing (NLP) have been driven bypretrained language models like BERT, RoBERTa, T5, and GPT. Thesemodels excel at understanding complex texts, but biomedical literature, withits domain-specific terminology, poses challenges that models likeWord2Vec and bidirectional long short-term memory (Bi-LSTM) can't fullyaddress. GPT and T5, despite capturing context, fall short in tasks needingbidirectional understanding, unlike BERT. Addressing this, we proposedMedicalBERT, a pretrained BERT model trained on a large biomedicaldataset and equipped with domain-specific vocabulary that enhances thecomprehension of biomedical terminology. MedicalBERT model is furtheroptimized and fine-tuned to address diverse tasks, including named entityrecognition, relation extraction, question answering, sentence similarity, anddocument classification. Performance metrics such as the F1-score,accuracy, and Pearson correlation are employed to showcase the efficiencyof our model in comparison to other BERT-based models such as BioBERT,SciBERT, and ClinicalBERT. MedicalBERT outperforms these models onmost of the benchmarks, and surpasses the general-purpose BERT model by5.67% on average across all the tasks evaluated respectively. This work alsounderscores the potential of leveraging pretrained BERT models for medicalNLP tasks, demonstrating the effectiveness of transfer learning techniques incapturing domain-specific information. (PDF) MedicalBERT: enhancing biomedical natural language processing using pretrained BERT-based model. Available from: this https URL [accessed Jul 06 2025].
Abstract (translated)
近期，自然语言处理（NLP）领域的进展主要归功于像BERT、RoBERTa、T5和GPT这样的预训练语言模型。这些模型在理解复杂文本方面表现出色，但生物医学文献因其领域特定的术语而提出了挑战，这是Word2Vec和双向长短时记忆网络（Bi-LSTM）等传统方法难以完全解决的问题。虽然GPT和T5能够捕捉上下文信息，但在需要双向理解的任务中不如BERT表现优秀。为此，我们提出了一种称为MedicalBERT的新模型，这是一种基于大量生物医学数据集训练的预训练BERT模型，并配备了领域特定词汇表以增强对生物医学术语的理解能力。通过进一步优化和微调，MedicalBERT能够在命名实体识别、关系抽取、问答、句子相似度和文档分类等多样化任务中发挥出色性能。 为了展示我们的模型相较于其他基于BERT的模型（如BioBERT、SciBERT和ClinicalBERT）在效率上的优势，我们采用了诸如F1分数、准确率以及Pearson相关系数等多种性能指标进行评估。实验结果表明，在大多数基准测试上，MedicalBERT均超越了这些模型，并且平均而言比通用型的BERT模型高出5.67%的表现（按任务分别计算）。这项工作不仅强调了利用预训练的BERT模型来处理医疗NLP任务的巨大潜力，还证明了迁移学习技术在捕捉领域特定信息方面的有效性。 论文《MedicalBERT：使用基于预训练BERT的模型增强生物医学自然语言处理》可以从以下链接下载：[此链接] [最后访问日期: 2025年7月6日]。
URL
https://arxiv.org/abs/2507.08013
