Abstract
There is substantial interest in developing artificial intelligence systems to support radiologists across tasks ranging from segmentation to report generation. Existing computed tomography (CT) foundation models have largely focused on building generalist vision-language systems capable of tasks such as question answering and report generation. However, training reliable vision-language systems requires paired image-text data at a scale that remains unavailable in CT. Moreover, adapting the underlying visual representations to downstream tasks typically requires partial or full backbone fine-tuning, a computationally demanding process inaccessible to many research groups. Instead, foundation models should prioritise learning robust visual representations that enable efficient transfer to new tasks with minimal labelled data and without backbone fine-tuning. We present VoxelFM, a 3D CT foundation model trained with self-distillation using the DINO framework, which learns semantically rich features without language supervision. We evaluated VoxelFM across seven categories of clinically relevant downstream tasks using frozen backbone representations with lightweight probes: classification, regression, survival analysis, instance retrieval, localisation, segmentation, and report generation. VoxelFM matched or outperformed four existing CT foundation models across all task categories. Despite receiving no language supervision during pre-training, VoxelFM surpassed models explicitly trained with language-alignment objectives, including on report generation. Our results indicate that current CT foundation models perform significantly better as feature extractors for lightweight probes rather than as vision encoders for vision-language models. Model weights and training code are publicly available.
Abstract (translated)
目前，医学界对开发人工智能系统以辅助放射科医生完成从分割到报告生成等任务抱有浓厚兴趣。现有的计算机断层扫描（CT）基础模型大多致力于构建能够执行问答和报告生成等任务的通用视觉-语言系统。然而，训练可靠的视觉-语言系统需要大规模配对的图像-文本数据，而CT领域目前仍缺乏此类数据。此外，将底层视觉表征适配到下游任务通常需要对主干网络进行部分或全部微调，这一计算密集型过程令许多研究团队难以企及。因此，基础模型应优先学习鲁棒的视觉表征，使其能够在无需微调主干网络、仅需少量标注数据的情况下高效迁移至新任务。本文提出VoxelFM，一个基于DINO框架通过自蒸馏训练的3D CT基础模型，可在无语言监督的情况下学习语义丰富的特征。我们使用冻结主干网络的特征表示结合轻量级探测头，在七个类别的临床相关下游任务上评估了VoxelFM：分类、回归、生存分析、实例检索、定位、分割和报告生成。结果显示，VoxelFM在所有任务类别中均达到或超越了四个现有CT基础模型的性能。尽管预训练阶段未接受语言监督，VoxelFM在报告生成等任务上仍超越了明确采用语言对齐目标训练的模型。我们的研究表明，当前CT基础模型作为轻量级探测头的特征提取器，其表现显著优于作为视觉-语言模型的视觉编码器。模型权重与训练代码已公开。
URL
https://arxiv.org/abs/2604.04133
