Abstract
Vision-language models (VLMs) have recently been integrated into multiple instance learning (MIL) frameworks to address the challenge of few-shot, weakly supervised classification of whole slide images (WSIs). A key trend involves leveraging multi-scale information to better represent hierarchical tissue structures. However, existing methods often face two key limitations: (1) insufficient modeling of interactions within the same modalities across scales (e.g., 5x and 20x) and (2) inadequate alignment between visual and textual modalities on the same scale. To address these gaps, we propose HiVE-MIL, a hierarchical vision-language framework that constructs a unified graph consisting of (1) parent-child links between coarse (5x) and fine (20x) visual/textual nodes to capture hierarchical relationships, and (2) heterogeneous intra-scale edges linking visual and textual nodes on the same scale. To further enhance semantic consistency, HiVE-MIL incorporates a two-stage, text-guided dynamic filtering mechanism that removes weakly correlated patch-text pairs, and introduces a hierarchical contrastive loss to align textual semantics across scales. Extensive experiments on TCGA breast, lung, and kidney cancer datasets demonstrate that HiVE-MIL consistently outperforms both traditional MIL and recent VLM-based MIL approaches, achieving gains of up to 4.1% in macro F1 under 16-shot settings. Our results demonstrate the value of jointly modeling hierarchical structure and multimodal alignment for efficient and scalable learning from limited pathology data. The code is available at this https URL
Abstract (translated)
最近，视觉-语言模型（VLMs）被整合到多实例学习（MIL）框架中，以解决全滑动图像（WSI）的少量样本、弱监督分类问题。一个关键的趋势是利用跨尺度的信息来更好地表示组织结构的层次性。然而，现有方法通常面临两个主要限制：(1) 在同一模态内（例如5x和20x）不同尺度之间的交互建模不足；(2) 同一尺度上视觉与文本模态间的对齐度不够。为解决这些问题，我们提出了HiVE-MIL，这是一个层次化视觉-语言框架，它构建了一个统一的图结构，其中包括：(1) 粗略（5x）和精细（20x）视觉/文本节点之间的父-子链接，以捕捉层次关系；以及 (2) 同一尺度上的异构内部边连接同一尺度下的视觉与文本节点。为了进一步增强语义一致性，HiVE-MIL引入了一种两阶段、由文本引导的动态过滤机制，用于移除相关性较弱的图块-文本对，并引入了跨尺度的层次对比损失来调整不同尺度上的文本语义。在TCGA乳腺癌、肺癌和肾癌数据集上进行的大量实验表明，HiVE-MIL始终优于传统的MIL方法以及近期基于VLM的MIL方法，在16个样本设置下实现了高达4.1%的宏F1值提升。我们的结果证明了同时建模层次结构和多模态对齐对于从有限病理学数据中进行高效、可扩展学习的价值。代码可在提供的链接处获取。
URL
https://arxiv.org/abs/2505.17982
