Abstract
Recent progress in large-scale CLIP-like vision-language models(VLMs) has greatly advanced medical image analysis. However, most existing medical VLMs still rely on coarse image-text contrastive objectives and fail to capture the systematic visual knowledge encoded in well-defined medical phenotype ontologies. To address this gap, we construct PhenoKG, the first large-scale, phenotype-centric multimodal knowledge graph that encompasses over 520K high-quality image-text pairs linked to more than 3,000 phenotypes. Building upon PhenoKG, we propose PhenoLIP, a novel pretraining framework that explicitly incorporates structured phenotype knowledge into medical VLMs through a two-stage process. We first learn a knowledge-enhanced phenotype embedding space from textual ontology data and then distill this structured knowledge into multimodal pretraining via a teacher-guided knowledge distillation objective. To support evaluation, we further introduce PhenoBench, an expert-verified benchmark designed for phenotype recognition, comprising over 7,800 image--caption pairs covering more than 1,000 phenotypes. Extensive experiments demonstrate that PhenoLIP outperforms previous state-of-the-art baselines, improving upon BiomedCLIP in phenotype classification accuracy by 8.85\% and BIOMEDICA in cross-modal retrieval by 15.03%, underscoring the value of integrating phenotype-centric priors into medical VLMs for structured and interpretable medical image understanding.
Abstract (translated)
近期在大规模类似CLIP的视觉-语言模型（VLMs）方面取得的进步极大地推动了医学图像分析的发展。然而，大多数现有的医疗VLM仍然依赖于粗略的图像文本对比目标，并且无法捕捉到在定义明确的医学表型本体论中编码的系统性视觉知识。为了填补这一空白，我们构建了PhenoKG，这是首个大规模、以表型为中心的多模态知识图谱，涵盖了超过520,000张高质量的图像-文本对，并且这些数据链接到了3,000多个不同的表型上。 基于PhenoKG，我们提出了一个新的预训练框架——PhenoLIP。这个框架通过一个两阶段过程明确地将结构化的表型知识整合到医疗VLMs中。首先，我们从文本本体数据中学得增强的表型嵌入空间，并且然后通过一种教师指导的知识蒸馏目标将其结构化知识浓缩进多模态预训练。 为了支持评估，我们进一步引入了PhenoBench，这是一个经过专家验证的基准测试集，专门用于表型识别，包含超过7,800张图像-描述对，涵盖了1,000多个不同的表型。大量的实验表明，PhenoLIP超越了之前最先进的基线模型，在表型分类准确率上比BiomedCLIP高出8.85%，在跨模态检索任务上比BIOMEDICA高出15.03%。这突显出将基于中心的先验知识整合到医疗VLMs中的重要性，以实现结构化和可解释的医学图像理解。
URL
https://arxiv.org/abs/2602.06184
