Abstract
Neural language models, particularly large-scale ones, have been consistently proven to be most effective in predicting brain neural activity across a range of studies. However, previous research overlooked the comparison of these models with psychologically plausible ones. Moreover, evaluations were reliant on limited, single-modality, and English cognitive datasets. To address these questions, we conducted an analysis comparing encoding performance of various neural language models and psychologically plausible models. Our study utilized extensive multi-modal cognitive datasets, examining bilingual word and discourse levels. Surprisingly, our findings revealed that psychologically plausible models outperformed neural language models across diverse contexts, encompassing different modalities such as fMRI and eye-tracking, and spanning languages from English to Chinese. Among psychologically plausible models, the one incorporating embodied information emerged as particularly exceptional. This model demonstrated superior performance at both word and discourse levels, exhibiting robust prediction of brain activation across numerous regions in both English and Chinese.
Abstract (translated)
神经语言模型，特别是大规模的 ones，在预测跨多个研究的脑神经活动方面一直被证明是最有效的。然而，之前的研究忽略了这些模型与心理上可解释的模型的比较。此外，评估是基于有限的、单模态的英语认知数据集进行的。为了回答这些问题，我们进行了一个比较各种神经语言模型和心理上可解释模型的编码性能的分析。我们的研究利用了广泛的跨模态认知数据集，研究了双语单词和话语水平。令人惊讶的是，我们的研究结果表明，心理上可解释的模型在不同的上下文中均优于神经语言模型，包括不同的模式，如 fMRI 和眼动，跨越了英语到汉语的语言。在心理上可解释的模型中，采用身体信息的模型表现得尤为出色。这个模型在词和 discourse 水平上表现出卓越的性能，展示了 robust 的预测 of brain activation across numerous regions in both English and Chinese.
URL
https://arxiv.org/abs/2404.19364
