Abstract
Vision language models (VLM) pre-trained on datasets of histopathological image-caption pairs enabled zero-shot slide-level classification. The ability of VLM image encoders to extract discriminative features also opens the door for supervised fine-tuning for whole-slide image (WSI) classification, ideally using few labeled samples. Slide-level prediction frameworks require the incorporation of multiple instance learning (MIL) due to the gigapixel size of the WSI. Following patch-level feature extraction and aggregation, MIL frameworks rely on linear classifiers trained on top of the slide-level aggregated features. Classifier weight initialization has a large influence on Linear Probing performance in efficient transfer learning (ETL) approaches based on few-shot learning. In this work, we propose Zero-Shot Multiple-Instance Learning (ZS-MIL) to address the limitations of random classifier initialization that underperform zero-shot prediction in MIL problems. ZS-MIL uses the class-level embeddings of the VLM text encoder as the classification layer's starting point to compute each sample's bag-level probabilities. Through multiple experiments, we demonstrate the robustness of ZS-MIL compared to well-known weight initialization techniques both in terms of performance and variability in an ETL few-shot scenario for subtyping prediction.
Abstract (translated)
基于病理图像-描述对数据集预训练的视觉语言模型（VLM）实现了零样本幻灯片级别分类。VLM图像编码器提取判别性特征的能力也为监督微调用于全滑片图像（WSI）分类打开了大门，理想情况下使用少量标注样本。由于WSI的千兆像素大小，幻灯片级别的预测框架需要融入多实例学习（MIL）。在基于补丁级别特征提取和聚合之后，MIL框架依赖于在幻灯片级别聚合特征上训练的线性分类器。分类权重初始化对基于少样本学习的有效迁移学习（ETL）方法中的线性探测性能有很大影响。在这项工作中，我们提出了零样本多实例学习（ZS-MIL），以解决随机分类器初始设置在MIL问题中表现不佳于零样本预测的局限性。ZS-MIL使用VLM文本编码器的类别级嵌入作为计算每个样本包级别概率的起始点。通过多个实验，我们展示了与已知权重初始化技术相比，在ETL少样本场景下子类型预测中的性能和变异性方面，ZS-MIL的鲁棒性。
URL
https://arxiv.org/abs/2602.18766
