Abstract
PURPOSE: Incident reports are an important tool for safety and quality improvement in healthcare, but manual review is time-consuming and requires subject matter expertise. Here we present a natural language processing (NLP) screening tool to detect high-severity incident reports in radiation oncology across two institutions. METHODS AND MATERIALS: We used two text datasets to train and evaluate our NLP models: 7,094 reports from our institution (Inst.), and 571 from IAEA SAFRON (SF), all of which had severity scores labeled by clinical content experts. We trained and evaluated two types of models: baseline support vector machines (SVM) and BlueBERT which is a large language model pretrained on PubMed abstracts and hospitalized patient data. We assessed for generalizability of our model in two ways. First, we evaluated models trained using Inst.-train on SF-test. Second, we trained a BlueBERT_TRANSFER model that was first fine-tuned on Inst.-train then on SF-train before testing on SF-test set. To further analyze model performance, we also examined a subset of 59 reports from our Inst. dataset, which were manually edited for clarity. RESULTS Classification performance on the Inst. test achieved AUROC 0.82 using SVM and 0.81 using BlueBERT. Without cross-institution transfer learning, performance on the SF test was limited to an AUROC of 0.42 using SVM and 0.56 using BlueBERT. BlueBERT_TRANSFER, which was fine-tuned on both datasets, improved the performance on SF test to AUROC 0.78. Performance of SVM, and BlueBERT_TRANSFER models on the manually curated Inst. reports (AUROC 0.85 and 0.74) was similar to human performance (AUROC 0.81). CONCLUSION: In summary, we successfully developed cross-institution NLP models on incident report text from radiation oncology centers. These models were able to detect high-severity reports similarly to humans on a curated dataset.
Abstract (translated)
目的：事故报告是医疗保健中安全和质量改进的重要工具，但手动审查耗时且需要专业知识。在此，我们介绍了一种自然语言处理（NLP）筛选工具，用于检测两个机构中的放射肿瘤学高严重性事故报告。 方法与材料：我们使用了两个文本数据集来训练和评估我们的NLP模型：来自我们机构的7,094份报告以及IAEA SAFRON(SF) 的571份报告，所有这些报告都已由临床内容专家进行了严重程度评分。我们训练并评估了两种类型的模型：基线支持向量机（SVM）和BlueBERT，这是一种在PubMed摘要和住院患者数据上进行预训练的大型语言模型。为了测试我们的模型的泛化能力，我们在两个方面进行了评估。首先，在使用Inst.-train训练后对SF-test进行了评估。其次，我们训练了BlueBERT_TRANSFER 模型，该模型先是在 Inst.-train 上进行微调，然后在 SF-train上进行微调，最后在SF测试集上进行测试。为了进一步分析模型性能，我们也检查了我们机构数据集中手动编辑以提高清晰度的59份报告的一个子集。 结果：在Inst. 测试上的分类表现使用SVM达到AUROC 0.82，使用BlueBERT 达到 AUROC 0.81。没有跨机构迁移学习的情况下，在SF测试中的性能仅限于 SVM 的AUROC为 0.42 和 BlueBERT的AUROC为 0.56。经过两个数据集微调后的BlueBERT_TRANSFER 模型，将 SF 测试上的表现提高到 AUROC 0.78。在手动整理的Inst. 报告上，SVM和BlueBERT_TRANSFER模型（AUROC分别为0.85和0.74）的表现与人类表现相似（AUROC为0.81）。 结论：总的来说，我们成功地开发了跨机构NLP模型用于放射肿瘤学中心的事故报告文本。这些模型能够在整理后的数据集中类似地检测出高严重性报告。
URL
https://arxiv.org/abs/2509.13706
