Abstract
Foundation models pre-trained on large-scale datasets demonstrate strong transfer learning capabilities; however, their adaptation to complex multi-label diagnostic tasks-such as comprehensive head CT finding detection-remains understudied. Standard parameter-efficient fine-tuning methods such as LoRA apply uniform adaptations across pathology types, which may limit performance for diverse medical findings. We propose a Mixture of Low-Rank Experts (MoLRE) framework that extends LoRA with multiple specialized low-rank adapters and unsupervised soft routing. This approach enables conditional feature adaptation with less than 0.5% additional parameters and without explicit pathology supervision. We present a comprehensive benchmark of MoLRE across six state-of-the-art medical imaging foundation models spanning 2D and 3D architectures, general-domain, medical-domain, and head CT-specific pretraining, and model sizes ranging from 7M to 431M parameters. Using over 70,000 non-contrast head CT scans with 75 annotated findings-including hemorrhage, infarction, trauma, mass lesions, structural abnormalities, and chronic changes-our experiments demonstrate consistent performance improvements across all models. Gains vary substantially: general-purpose and medical-domain models show the largest improvements (DINOv3-Base: +4.6%; MedGemma: +4.3%), whereas 3D CT-specialized or very large models show more modest gains (+0.2-1.3%). The combination of MoLRE and MedGemma achieves the highest average detection AUC of 0.917. These findings highlight the importance of systematic benchmarking on target clinical tasks, as pretraining domain, architecture, and model scale interact in non-obvious ways.
Abstract (translated)
预训练于大规模数据集上的基础模型展示了强大的迁移学习能力；然而，它们在复杂多标签诊断任务（如全面的头部CT发现检测）中的适应性研究仍然不足。标准的参数高效微调方法，例如LoRA（低秩适应），对各种病理类型应用统一的适应策略，这可能限制了多样医学发现性能的提升。我们提出了一种“混合低秩专家”(Mixture of Low-Rank Experts, MoLRE)框架，该框架通过引入多个专门化的低秩适配器和无监督软路由来扩展LoRA方法。这种方法能够在不增加额外参数（少于0.5%）且无需明确的病理学指导的情况下实现条件特征适应。 我们对MoLRE进行了全面的基准测试，涉及六种最先进的医学成像基础模型，这些模型涵盖了二维和三维架构、通用领域和医疗领域的预训练以及大小从7M到431M参数的不同规模。使用超过70,000个非对比头部CT扫描（包含75种标注发现，如出血、梗塞、创伤、肿块病变、结构异常及慢性改变），我们的实验在所有模型上均展示了持续的性能提升。收益差异显著：通用型和医疗领域模型显示了最大的改进（DINOv3-Base: +4.6%；MedGemma: +4.3%），而三维CT专门化或非常大规模的模型则表现出较为温和的增长(+0.2-1.3%)。 MoLRE与MedGemma的结合实现了最高的平均检测AUC值为0.917。这些发现强调了在目标临床任务上进行系统基准测试的重要性，因为预训练领域、架构和模型规模之间的相互作用以非显而易见的方式存在。
URL
https://arxiv.org/abs/2603.00675
