Abstract
Foundation models for medical imaging are typically pretrained on increasingly large datasets, following a "scale-at-all-costs" paradigm. However, this strategy faces two critical challenges: large-scale medical datasets often contain substantial redundancy and severe class imbalance that bias representation learning toward over-represented patterns, and indiscriminate training regardless of heterogeneity in data quality incurs considerable computational inefficiency. Here we demonstrate that active, principled data curation during pretraining can serve as a viable, cost-effective alternative to brute-force dataset enlargement. We introduce CheXficient, a chest X-ray (CXR) foundation model that selectively prioritizes informative training samples. CheXficient is pretrained on only 22.7% of 1,235,004 paired CXR images and reports while consuming under 27.3% of the total compute budget, yet achieving comparable or superior performance to its full-data counterpart and other large-scale pretrained models. We assess CheXficient across 20 individual benchmarks spanning 5 task types, including non-adapted off-the-shelf evaluations (zero-shot findings classification and crossmodal retrieval) and adapted downstream tasks (disease prediction, semantic segmentation, and radiology report generation). Further analyses show that CheXficient systematically prioritizes under-represented training samples, improving generalizability on long-tailed or rare conditions. Overall, our work offers practical insights into the data and computation demands for efficient pretraining and downstream adaptation of medical vision-language foundation models.
Abstract (translated)
医学成像的基础模型通常在越来越大的数据集上进行预训练，遵循“不惜一切代价扩大规模”的理念。然而，这种策略面临两个关键挑战：大规模的医疗数据集中往往包含大量的冗余和严重的类别不平衡，这会导致表征学习偏向于过度表示的模式；此外，在不考虑数据质量异质性的情况下无差别地进行训练会带来相当大的计算低效性。在这里，我们证明了在预训练期间采取主动、有原则的数据管理可以作为单纯扩大数据集规模的一种可行且成本效益更高的替代方案。 为此，我们引入了一种胸部X光（CXR）基础模型——CheXficient，该模型选择性地优先考虑具有信息量的训练样本。CheXficient仅在1,235,004对配对的CXR图像和报告中的22.7%上进行预训练，并且计算资源消耗不到总预算的27.3%，但其性能与全数据集版本和其他大规模预训练模型相当或更优。我们通过对包括零样本发现分类、跨模态检索（未适应现成评估）、疾病预测、语义分割和放射报告生成在内的五类任务类型中的二十个独立基准进行评测，来评估CheXficient的性能。 进一步分析表明，CheXficient系统地优先考虑代表性较低的训练样本，在长尾或罕见条件下具有更好的泛化能力。总体而言，我们的工作为医学视觉-语言基础模型的有效预训练和下游任务适应的数据及计算需求提供了实用见解。
URL
https://arxiv.org/abs/2602.22843
