Abstract
Vision-language pre-training (VLP) has great potential for developing multifunctional and general medical diagnostic capabilities. However, aligning medical images with a low signal-to-noise ratio (SNR) to reports with a high SNR presents a semantic density gap, leading to visual alignment bias. In this paper, we propose boosting vision semantic density to improve alignment effectiveness. On one hand, we enhance visual semantics through disease-level vision contrastive learning, which strengthens the model's ability to differentiate between normal and abnormal samples for each anatomical structure. On the other hand, we introduce an anatomical normality modeling method to model the distribution of normal samples for each anatomy, leveraging VQ-VAE for reconstructing normal vision embeddings in the latent space. This process amplifies abnormal signals by leveraging distribution shifts in abnormal samples, enhancing the model's perception and discrimination of abnormal attributes. The enhanced visual representation effectively captures the diagnostic-relevant semantics, facilitating more efficient and accurate alignment with the diagnostic report. We conduct extensive experiments on two chest CT datasets, CT-RATE and Rad-ChestCT, and an abdominal CT dataset, MedVL-CT69K, and comprehensively evaluate the diagnosis performance across multiple tasks in the chest and abdominal CT scenarios, achieving state-of-the-art zero-shot performance. Notably, our method achieved an average AUC of 84.9% across 54 diseases in 15 organs, significantly surpassing existing methods. Additionally, we demonstrate the superior transfer learning capabilities of our pre-trained model. Code is available at this https URL.
Abstract (translated)
视觉语言预训练（VLP）在开发多功能和通用的医学诊断能力方面具有巨大潜力。然而，将信号与噪声比（SNR）较低的医疗图像与SNR较高的报告对齐存在语义密度差距问题，导致视觉对齐偏差。本文中，我们提出了一种通过增强视觉语义密度来提高对齐效果的方法。一方面，我们通过疾病级别的视觉对比学习增强了视觉语义，这使模型能够更好地区分每个解剖结构的正常和异常样本的能力。另一方面，我们引入了解剖学正常性建模方法，为每一种解剖构造中的正常样本分布建模，并利用VQ-VAE在潜在空间中重构正常的视觉嵌入。这个过程通过利用异常样本中的分布变化来放大异常信号，从而增强了模型对异常属性的感知和识别能力。增强后的视觉表示有效地捕捉了与诊断相关的语义信息，有助于更高效、准确地将其与诊断报告进行对齐。 我们在两个胸部CT数据集（CT-RATE 和 Rad-ChestCT）以及一个腹部CT数据集（MedVL-CT69K）上进行了广泛的实验，并全面评估了多种任务中的诊断性能，在胸部和腹部CT场景中实现了最先进的零样本性能。值得注意的是，我们的方法在15个器官的54种疾病上的平均AUC达到84.9%，显著超过了现有方法的表现。此外，我们展示了预训练模型优越的迁移学习能力。 代码可在以下链接获取：[此 URL](https://this https URL)（请将“this https URL”替换为实际提供的链接地址）。
URL
https://arxiv.org/abs/2508.03742
