Abstract
The rapid advancement of large-scale vision-language models has showcased remarkable capabilities across various tasks. However, the lack of extensive and high-quality image-text data in medicine has greatly hindered the development of large-scale medical vision-language models. In this work, we present a diagnosis-guided bootstrapping strategy that exploits both image and label information to construct vision-language datasets. Based on the constructed dataset, we developed MedDr, a generalist foundation model for healthcare capable of handling diverse medical data modalities, including radiology, pathology, dermatology, retinography, and endoscopy. Moreover, during inference, we propose a simple but effective retrieval-augmented medical diagnosis strategy, which enhances the model's generalization ability. Extensive experiments on visual question answering, medical report generation, and medical image diagnosis demonstrate the superiority of our method.
Abstract (translated)
大规模视觉语言模型的快速发展在各种任务中展示了令人印象深刻的性能。然而，在医学领域中缺乏大量高质量的图像-文本数据，大大阻碍了大规模医疗视觉语言模型的开发。在这项工作中，我们提出了一个指导下的bootstrap策略，该策略利用图像和标签信息来构建视觉语言数据集。基于构建的数据集，我们开发了MedDr，一种通用医疗数据处理模型，能够处理各种医疗数据模式，包括放射学、病理学、皮肤病学、眼科和内窥镜。此外，在推理过程中，我们提出了一种简单但有效的检索增强医疗诊断策略，可以增强模型的泛化能力。在视觉问答、医学报告生成和医学图像诊断等大量实验中，证明了我们方法的优势。
URL
https://arxiv.org/abs/2404.15127
