Abstract
AI-driven clinical text classification is vital for explainable automated retrieval of population-level health information. This work investigates whether human-based clinical rationales can serve as additional supervision to improve both performance and explainability of transformer-based models that automatically encode clinical documents. We analyzed 99,125 human-based clinical rationales that provide plausible explanations for primary cancer site diagnoses, using them as additional training samples alongside 128,649 electronic pathology reports to evaluate transformer-based models for extracting primary cancer sites. We also investigated sufficiency as a way to measure rationale quality for pre-selecting rationales. Our results showed that clinical rationales as additional training data can improve model performance in high-resource scenarios but produce inconsistent behavior when resources are limited. Using sufficiency as an automatic metric to preselect rationales also leads to inconsistent results. Importantly, models trained on rationales were consistently outperformed by models trained on additional reports instead. This suggests that clinical rationales don't consistently improve model performance and are outperformed by simply using more reports. Therefore, if the goal is optimizing accuracy, annotation efforts should focus on labeling more reports rather than creating rationales. However, if explainability is the priority, training models on rationale-supplemented data may help them better identify rationale-like features. We conclude that using clinical rationales as additional training data results in smaller performance improvements and only slightly better explainability (measured as average token-level rationale coverage) compared to training on additional reports.
Abstract (translated)
基于AI的临床文本分类对于自动检索人群水平健康信息至关重要，并且能够提供可解释性。这项研究探讨了人类生成的临床理由是否可以作为额外监督数据，以提高基于Transformer模型在自动编码临床文档方面性能和可解释性的效果。我们分析了99,125个人类生成的临床理由，这些理由为原发肿瘤部位诊断提供了合理的解释，并将其用作训练样本，与128,649份电子病理报告一起评估用于提取原发癌部位的Transformer模型。此外，我们还研究了充足性作为选择理性质量的一种方式来预先挑选理性的方法。 我们的结果显示，在高资源场景下，将临床理由作为额外的数据进行训练可以提高模型性能，但在资源有限的情况下则会产生不一致的行为。使用充分性作为自动度量标准预选合理性同样会导致结果的不一致性。重要的是，基于合理性的模型始终被额外报告训练出的模型所超越。这表明临床理性并不一贯地提升模型表现，并且在增加更多的报告时会被超越。 因此，如果目标是优化准确率，注解工作应该集中在标记更多报告上而不是创建合理性解释。然而，如果优先考虑的是可解释性，基于补充了理性的数据训练模型可能有助于它们更好地识别类似理由的特征。 综上所述，使用临床理性作为额外的数据进行训练仅在性能改进方面提供较小的优势，并且相对于增加更多报告而言，在提高可解释性（以平均令牌级合理性覆盖率衡量）方面的优势也微乎其微。
URL
https://arxiv.org/abs/2507.21302
