Abstract
Medical vision-language pre-training has emerged as a promising approach for learning domain-general representations of medical image and text. Current algorithms that exploit the global and local alignment between medical image and text could however be marred by the redundant information in medical data. To address this issue, we propose a grounded knowledge-enhanced medical vision-language pre-training (GK-MVLP) framework for chest X-ray. In this framework, medical knowledge is grounded to the appropriate anatomical regions by using a transformer-based grounded knowledge-enhanced module for fine-grained alignment between anatomical region-level visual features and the textural features of medical knowledge. The performance of GK-MVLP is competitive with or exceeds the state of the art on downstream chest X-ray disease classification, disease localization, report generation, and medical visual question-answering tasks. Our results show the advantage of incorporating grounding mechanism to remove biases and improve the alignment between chest X-ray image and radiology report.
Abstract (translated)
医学视觉语言预训练被证明是一种学习医学图像和文本领域通用表示的有前途的方法。然而,当前的算法可能受到医疗数据中冗余信息的影响。为了应对这个问题,我们提出了一个基于知识增强的医学视觉语言预训练(GK-MVLP)框架,用于胸部X光片。在这个框架中,医学知识是通过使用基于Transformer的 grounded knowledge-enhanced 模块将解剖区域级别的视觉特征与医学知识的文本特征进行精细对齐来 grounded 的。GK-MVLP 在下游胸部X光片疾病分类、疾病定位、报告生成和医疗视觉问答等任务上的性能与或超过最先进的水平。我们的结果表明,将校准机制集成到系统中可以消除偏见并改善胸部X光片图像与放射科报告之间的对齐。
URL
https://arxiv.org/abs/2404.14750
