Abstract
Domain-specific foundation models for healthcare have expanded rapidly in recent years, yet foundation models for critical care time series remain relatively underexplored due to the limited size and availability of datasets. In this work, we introduce an early-stage pre-trained foundation model for critical care time-series based on the Bi-Axial Transformer (BAT), trained on pooled electronic health record datasets. We demonstrate effective transfer learning by fine-tuning the model on a dataset distinct from the training sources for mortality prediction, where it outperforms supervised baselines, particularly for small datasets ($<5,000$). These contributions highlight the potential of self-supervised foundation models for critical care times series to support generalizable and robust clinical applications in resource-limited settings.
Abstract (translated)
近年来，医疗领域的特定领域基础模型发展迅速，然而由于数据集规模和可用性的限制，针对重症监护时间序列的基础模型仍然相对较少被探索。在这项工作中，我们引入了一个基于双轴变换器（Bi-Axial Transformer, BAT）的早期预训练基础模型，并在汇总的电子健康记录数据集上进行了训练。通过将该模型微调应用于与训练源不同的数据集来进行死亡率预测，我们展示了有效的迁移学习能力，其性能优于监督基线模型，尤其是在小型数据集（少于5000个样本）的情况下。这些贡献突显了自我监督基础模型在重症监护时间序列中的潜力，能够支持资源有限环境下的通用性和稳健性的临床应用。
URL
https://arxiv.org/abs/2509.19885
