Abstract
Introduction: Clinical text classification using natural language processing (NLP) models requires adequate training data to achieve optimal performance. For that, 200-500 documents are typically annotated. The number is constrained by time and costs and lacks justification of the sample size requirements and their relationship to text vocabulary properties. Methods: Using the publicly available MIMIC-III dataset containing hospital discharge notes with ICD-9 diagnoses as labels, we employed pre-trained BERT embeddings followed by Random Forest classifiers to identify 10 randomly selected diagnoses, varying training corpus sizes from 100 to 10,000 documents, and analyzed vocabulary properties by identifying strong and noisy predictive words through Lasso logistic regression on bag-of-words embeddings. Results: Learning curves varied significantly across the 10 classification tasks despite identical preprocessing and algorithms, with 600 documents sufficient to achieve 95% of the performance attainable with 10,000 documents for all tasks. Vocabulary analysis revealed that more strong predictors and fewer noisy predictors were associated with steeper learning curves, where every 100 additional noisy words decreased accuracy by approximately 0.02 while 100 additional strong predictors increased maximum accuracy by approximately 0.04.
Abstract (translated)
引言：使用自然语言处理（NLP）模型进行临床文本分类需要充足的训练数据以达到最佳性能。通常情况下，这要求对大约200到500份文档进行注释。然而，这个数字受到时间和成本的限制，并且缺乏关于样本量需求及其与文本词汇属性关系的合理解释。 方法：我们使用了公开可用的MIMIC-III数据集，该数据集中包含医院出院记录及相应的ICD-9诊断标签。通过应用预训练的BERT嵌入并随后采用随机森林分类器来识别10个随机选择的诊断，并且在从100到10,000份文档不等的训练语料库大小下进行了实验。此外，我们还利用bag-of-words（词袋）模型中的Lasso逻辑回归分析词汇属性，通过识别强预测词和噪声预测词来实现这一目标。 结果：尽管预处理过程与算法保持一致，但针对10个分类任务的学习曲线差异显著。对于所有任务而言，使用600份文档即可达到使用10,000份文档所能获得的95%性能水平。词汇分析揭示了更多的强预测词和更少的噪声预测词与陡峭的学习曲线之间的关联性：每增加100个噪声单词会使准确率降低约0.02，而每增加100个强预测词则会将最大准确度提高大约0.04。
URL
https://arxiv.org/abs/2601.15846
