Abstract
Medical vision-language models (VLMs) have demonstrated unprecedented transfer capabilities and are being increasingly adopted for data-efficient image classification. Despite its growing popularity, its reliability aspect remains largely unexplored. This work explores the split conformal prediction (SCP) framework to provide trustworthiness guarantees when transferring such models based on a small labeled calibration set. Despite its potential, the generalist nature of the VLMs' pre-training could negatively affect the properties of the predicted conformal sets for specific tasks. While common practice in transfer learning for discriminative purposes involves an adaptation stage, we observe that deploying such a solution for conformal purposes is suboptimal since adapting the model using the available calibration data breaks the rigid exchangeability assumptions for test data in SCP. To address this issue, we propose transductive split conformal adaptation (SCA-T), a novel pipeline for transfer learning on conformal scenarios, which performs an unsupervised transductive adaptation jointly on calibration and test data. We present comprehensive experiments utilizing medical VLMs across various image modalities, transfer tasks, and non-conformity scores. Our framework offers consistent gains in efficiency and conditional coverage compared to SCP, maintaining the same empirical guarantees.
Abstract (translated)
医学视觉-语言模型（VLMs）已经展示了前所未有的转移能力，并且越来越多地被用于基于少量标注数据进行高效的图像分类。尽管其流行度日益增加，但其可靠性方面仍鲜有研究。这项工作探索了分裂置信预测（SCP）框架，在将这些模型应用于特定任务时，利用一个小的标记校准集提供可信度保证。尽管具有潜在优势，VLMs预训练的一般特性可能会对特定任务的预测置信集合属性产生负面影响。虽然在转移学习中为了区分目的通常涉及一个适应阶段，但我们观察到将其用于置信性目的效果不佳，因为在使用可用校准数据调整模型时会破坏SCP中关于测试数据的刚性可交换性假设。 为了解决这个问题，我们提出了一种新型的管道——转导分裂置信预测适配（SCA-T），专门针对置信场景下的转移学习。该方法在标记和未标记数据上执行无监督的转导适应过程。我们在各种医学VLMs、不同的图像模态、转移任务以及非一致性评分上进行了全面实验，验证了我们框架的有效性。 我们的框架相比于SCP，在效率和条件覆盖率方面表现出一致的优势，并保持相同的实证保证。
URL
https://arxiv.org/abs/2506.17503
