Abstract
The global demand for radiologists is increasing rapidly due to a growing reliance on medical imaging services, while the supply of radiologists is not keeping pace. Advances in computer vision and image processing technologies present significant potential to address this gap by enhancing radiologists' capabilities and improving diagnostic accuracy. Large language models (LLMs), particularly generative pre-trained transformers (GPTs), have become the primary approach for understanding and generating textual data. In parallel, vision transformers (ViTs) have proven effective at converting visual data into a format that LLMs can process efficiently. In this paper, we present ChestGPT, a deep-learning framework that integrates the EVA ViT with the Llama 2 LLM to classify diseases and localize regions of interest in chest X-ray images. The ViT converts X-ray images into tokens, which are then fed, together with engineered prompts, into the LLM, enabling joint classification and localization of diseases. This approach incorporates transfer learning techniques to enhance both explainability and performance. The proposed method achieved strong global disease classification performance on the VinDr-CXR dataset, with an F1 score of 0.76, and successfully localized pathologies by generating bounding boxes around the regions of interest. We also outline several task-specific prompts, in addition to general-purpose prompts, for scenarios radiologists might encounter. Overall, this framework offers an assistive tool that can lighten radiologists' workload by providing preliminary findings and regions of interest to facilitate their diagnostic process.
Abstract (translated)
全球对放射科医生的需求由于医疗成像服务的依赖日益增加而迅速增长，然而放射科医生的数量并未跟上这一需求的步伐。计算机视觉和图像处理技术的进步为缓解这一缺口提供了巨大的潜力，能够增强放射科医生的能力并提高诊断准确性。大型语言模型（LLMs），尤其是生成式预训练变换器（GPTs），已经成为理解和生成文本数据的主要方法。同时，视觉变换器（ViTs）已被证明能够有效将视觉数据转换成适合LLMs高效处理的格式。在这篇论文中，我们提出了ChestGPT，这是一个深度学习框架，它结合了EVA ViT和Llama 2 LLM来对胸部X光片进行疾病分类及定位感兴趣的区域。ViT将X射线图像转化为令牌，然后与工程化的提示一起输入LLM，从而实现疾病的联合分类和定位。这种方法通过采用迁移学习技术增强了可解释性和性能。我们提出的方法在VinDr-CXR数据集上的全球疾病分类任务中表现出色，取得了0.76的F1分数，并成功地生成了围绕感兴趣区域的边界框来定位病灶。此外，我们也为放射科医生可能遇到的各种场景提供了几种特定任务提示和通用提示。总体而言，该框架提供了一种辅助工具，通过提供初步发现及感兴趣的区域来减轻放射科医生的工作负担，从而方便他们的诊断过程。
URL
https://arxiv.org/abs/2507.03739
