Abstract
Chest X-rays (CXRs) are the most frequently performed imaging test in clinical practice. Recent advances in the development of vision-language foundation models (FMs) give rise to the possibility of performing automated CXR interpretation, which can assist physicians with clinical decision-making and improve patient outcomes. However, developing FMs that can accurately interpret CXRs is challenging due to the (1) limited availability of large-scale vision-language datasets in the medical image domain, (2) lack of vision and language encoders that can capture the complexities of medical data, and (3) absence of evaluation frameworks for benchmarking the abilities of FMs on CXR interpretation. In this work, we address these challenges by first introducing \emph{CheXinstruct} - a large-scale instruction-tuning dataset curated from 28 publicly-available datasets. We then present \emph{CheXagent} - an instruction-tuned FM capable of analyzing and summarizing CXRs. To build CheXagent, we design a clinical large language model (LLM) for parsing radiology reports, a vision encoder for representing CXR images, and a network to bridge the vision and language modalities. Finally, we introduce \emph{CheXbench} - a novel benchmark designed to systematically evaluate FMs across 8 clinically-relevant CXR interpretation tasks. Extensive quantitative evaluations and qualitative reviews with five expert radiologists demonstrate that CheXagent outperforms previously-developed general- and medical-domain FMs on CheXbench tasks. Furthermore, in an effort to improve model transparency, we perform a fairness evaluation across factors of sex, race and age to highlight potential performance disparities. Our project is at \url{this https URL}.
Abstract (translated)
翻译 胸部X光（CXRs）是临床实践中最常进行的影像学检查。近年来，在视觉语言模型（FMs）的发展中，自动进行CXR interpretation的可能性得以实现，这可以协助医生进行临床决策并提高患者治疗效果。然而，开发可以准确解释CXR的FM具有挑战性，因为（1）在医学图像领域大型视觉语言数据集的可用性有限，（2）缺乏可以捕捉医学数据复杂性的视觉和语言编码器，（3）缺乏评估FM在CXR解释能力方面的标准框架。在这项工作中，我们通过首先介绍 CheXinstruct - 一个来自28个公开可用数据集的大型指令调整数据集来解决这些挑战。然后，我们介绍了 CheXagent - 一个经过指令调整的FM，可以分析和总结CXR。为了构建CheXagent，我们设计了一个用于解析放射学报告的临床大型语言模型、一个用于表示CXR图像的视觉编码器和一种桥接视觉和语言模型的网络。最后，我们介绍了 CheXbench - 一个旨在系统地评估FM在8个临床相关CXR解释任务上的新颖基准。通过对五个专家放射医生的深入定量评估和定性审查，我们可以得出结论，CheXagent在CheXbench任务上优于之前开发的一般和医学领域FM。此外，为了提高模型透明度，我们对性别、种族和年龄等因素进行了公平评估，以突出可能存在的性能差异。我们的项目位于此链接。
URL
https://arxiv.org/abs/2401.12208
