Abstract
A fundamental challenge in neuroscience is to decode mental states from brain activity. While functional magnetic resonance imaging (fMRI) offers a non-invasive approach to capture brain-wide neural dynamics with high spatial precision, decoding from fMRI data -- particularly from task-evoked activity -- remains challenging due to its high dimensionality, low signal-to-noise ratio, and limited within-subject data. Here, we leverage recent advances in computer vision and propose STDA-SwiFT, a transformer-based model that learns transferable representations from large-scale fMRI datasets via spatial-temporal divided attention and self-supervised contrastive learning. Using pretrained voxel-wise representations from 995 subjects in the Human Connectome Project (HCP), we show that our model substantially improves downstream decoding performance of task-evoked activity across multiple sensory and cognitive domains, even with minimal data preprocessing. We demonstrate performance gains from larger receptor fields afforded by our memory-efficient attention mechanism, as well as the impact of functional relevance in pretraining data when fine-tuning on small samples. Our work showcases transfer learning as a viable approach to harness large-scale datasets to overcome challenges in decoding brain activity from fMRI data.
Abstract (translated)
神经科学的一个基本挑战是从脑活动解码心理状态。虽然功能性磁共振成像（fMRI）提供了一种非侵入性的方法，能够以高空间精度捕捉全脑的神经动态变化，但从fMRI数据中进行解码——特别是从任务诱发的活动中解码——仍然具有挑战性，原因包括其维度高、信噪比低以及受试者间的数据量有限。在这里，我们借鉴了计算机视觉领域的最新进展，并提出了STDA-SwiFT模型，这是一种基于变压器架构的模型，通过空间-时间分解注意力和自我监督对比学习的方法从大规模fMRI数据集中学习可迁移表示。 利用来自人类连接组项目（HCP）中995名受试者的预训练体素级表示，我们展示了我们的模型在解码任务诱发活动方面性能显著提升，尤其是在多个感觉和认知领域，并且即使进行最少的数据预处理亦如此。我们还表明了更大的感受野如何得益于我们记忆高效的注意力机制而带来的性能改进，以及当在小样本上微调时，预训练数据中的功能相关性的影响。 我们的研究展示了迁移学习作为一种可行的方法，可以利用大规模数据集来克服从fMRI数据解码脑活动的挑战。
URL
https://arxiv.org/abs/2507.22378
