Abstract
Attention-based models have become the leading approach in modeling medical language for Natural Language Processing (NLP) in clinical notes. These models outperform traditional techniques by effectively capturing contextual rep- resentations of language. In this research a comparative analysis is done amongst pre- trained attention based models namely Bert Base, BioBert, two variations of Bio+Clinical Bert, RoBerta, and Clinical Long- former on task related to Electronic Health Record (EHR) information extraction. The tasks from Track 1 of Harvard Medical School's 2022 National Clinical NLP Challenges (n2c2) are considered for this comparison, with the Contextualized Medication Event Dataset (CMED) given for these task. CMED is a dataset of unstructured EHRs and annotated notes that contain task relevant information about the EHRs. The goal of the challenge is to develop effective solutions for extracting contextual information related to patient medication events from EHRs using data driven methods. Each pre-trained model is fine-tuned and applied on CMED to perform medication extraction, medical event detection, and multi-dimensional medication event context classification. Pro- cessing methods are also detailed for breaking down EHRs for compatibility with the applied models. Performance analysis has been carried out using a script based on constructing medical terms from the evaluation portion of CMED with metrics including recall, precision, and F1-Score. The results demonstrate that models pre-trained on clinical data are more effective in detecting medication and medication events, but Bert Base, pre- trained on general domain data showed to be the most effective for classifying the context of events related to medications.
Abstract (translated)
基于注意力的模型已经成为自然语言处理（NLP）在临床笔记中建模医学语言的领先方法。这些模型通过有效捕捉语言的上下文表示，超越了传统的技术。在这项研究中，对几个预训练的注意力模型进行比较分析，包括Bert Base、BioBert、两种不同的Bio+Clinical Bert变体、RoBerta和Clinical LongFormer，在与电子健康记录（EHR）信息提取相关的任务上进行了对比。这些任务来自哈佛医学院2022年全国临床NLP挑战赛（n2c2）的Track 1，使用的数据集是上下文化的药物事件数据集（CMED），该数据集包含未结构化和标注良好的EHR注释，其中包含了与EHR相关的任务相关信息。比赛的目标是开发有效的方法从EHR中提取患者用药情况的相关信息，并采用基于数据驱动的方法。 每个预训练模型都在CMED上进行了微调并应用于药物提取、医疗事件检测以及多维度的药物事件上下文分类。还详细说明了处理方法，用于分解EHR以使其与应用模型兼容。性能分析使用了一个基于从CMED评估部分构建医学术语的脚本进行，该脚本采用了包括召回率（recall）、精确度（precision）和F1分数在内的指标。 结果显示，在检测药物及药物事件方面，经过临床数据预训练的模型更有效；然而，在分类与药物相关的事件上下文时，通用领域数据上预训练的Bert Base被证明是最有效的。
URL
https://arxiv.org/abs/2509.19224
