Abstract
Understanding how the brain responds to external stimuli and decoding this process has been a significant challenge in neuroscience. While previous studies typically concentrated on brain-to-image and brain-to-language reconstruction, our work strives to reconstruct gestures associated with speech stimuli perceived by brain. Unfortunately, the lack of paired \{brain, speech, gesture\} data hinders the deployment of deep learning models for this purpose. In this paper, we introduce a novel approach, \textbf{fMRI2GES}, that allows training of fMRI-to-gesture reconstruction networks on unpaired data using \textbf{Dual Brain Decoding Alignment}. This method relies on two key components: (i) observed texts that elicit brain responses, and (ii) textual descriptions associated with the gestures. Then, instead of training models in a completely supervised manner to find a mapping relationship among the three modalities, we harness an fMRI-to-text model, a text-to-gesture model with paired data and an fMRI-to-gesture model with unpaired data, establishing dual fMRI-to-gesture reconstruction patterns. Afterward, we explicitly align two outputs and train our model in a self-supervision way. We show that our proposed method can reconstruct expressive gestures directly from fMRI recordings. We also investigate fMRI signals from different ROIs in the cortex and how they affect generation results. Overall, we provide new insights into decoding co-speech gestures, thereby advancing our understanding of neuroscience and cognitive science.
Abstract (translated)
理解大脑对外部刺激的反应并解码这一过程一直是神经科学面临的重大挑战。以往的研究通常集中在脑影像到图像或脑影像到语言的重建上，而我们的工作则致力于重构与由大脑感知的语言刺激相关的手势。遗憾的是，缺乏成对的{\{大脑, 语音, 手势\}}数据阻碍了深度学习模型在这一领域的部署。在这篇论文中，我们提出了一种新方法**fMRI2GES**，它允许使用**双脑解码对齐**技术在非配对的数据上训练fMRI到手势的重构网络。该方法依赖于两个关键组成部分：（i）引发大脑反应的观察文本；和（ii）与手势相关的文字描述。我们不是以完全监督的方式训练模型来寻找三种模式之间的映射关系，而是利用一个fMRI到文本的模型、一个使用配对数据的文字到手势模型以及一个基于非配对数据的fMRI到手势模型，建立双fMRI到手势重构模式。之后，我们将两个输出明确对齐并以自监督方式训练我们的模型。我们展示了所提出的方法可以直接从fMRI记录中重构出富有表现力的手势，并研究了皮层不同脑区（ROI）的fMRI信号及其对手部生成结果的影响。总体而言，我们为解码伴随语言的动作提供了新的见解，从而推进了对神经科学和认知科学的理解。
URL
https://arxiv.org/abs/2512.01189
