Abstract
In-context learning (ICL), a type of universal model, demonstrates exceptional generalization across a wide range of tasks without retraining by leveraging task-specific guidance from context, making it particularly effective for the intricate demands of neuroimaging. However, current ICL models, limited to 2D inputs and thus exhibiting suboptimal performance, struggle to extend to 3D inputs due to the high memory demands of ICL. In this regard, we introduce Neuroverse3D, an ICL model capable of performing multiple neuroimaging tasks in 3D (e.g., segmentation, denoising, inpainting). Neuroverse3D overcomes the large memory consumption associated with 3D inputs through adaptive parallel-sequential context processing and a U-shaped fusion strategy, allowing it to handle an unlimited number of context images. Additionally, we propose an optimized loss function to balance multi-task training and enhance focus on anatomical boundaries. Our study incorporates 43,674 3D multi-modal scans from 19 neuroimaging datasets and evaluates Neuroverse3D on 14 diverse tasks using held-out test sets. The results demonstrate that Neuroverse3D significantly outperforms existing ICL models and closely matches task-specific models, enabling flexible adaptation to medical center variations without retraining. The code and model weights are publicly available at this https URL.
Abstract (translated)
**翻译** 在上下文学习（ICL）这种通用模型中，通过利用任务特定的指导信息进行泛化处理，在不重新训练的情况下能够在广泛的任务范围内表现出色。特别是在神经成像的复杂需求方面尤为有效。然而，目前的ICL模型仅限于2D输入，因此性能表现不足，并且难以扩展到3D输入，因为这会带来巨大的内存消耗问题。鉴于此，我们引入了Neuroverse3D，这是一种能够执行多种神经成像任务（例如分割、去噪和填充）的ICL模型，在3D环境中运作。 通过自适应并行-串行上下文处理以及U形融合策略，Neuroverse3D克服了与3D输入相关的大量内存消耗问题，使其可以处理无限数量的上下文图像。此外，我们还提出了一种优化过的损失函数来平衡多任务训练，并增强对解剖边界的关注。 我们的研究采用了来自19个神经成像数据集中的43,674个3D多模态扫描，并在包含14项不同任务的保留测试集合上评估了Neuroverse3D的表现。结果表明，Neuroverse3D显著优于现有的ICL模型，并且接近于特定任务模型的效果，这使得它能够灵活地适应医疗中心之间的差异而不必重新训练。 该代码和模型权重可在以下网址公开获取：[https://this-url.com](https://this-url.com)
URL
https://arxiv.org/abs/2503.02410
