Abstract
Electroencephalography-to-Text generation (EEG-to-Text), which aims to directly generate natural text from EEG signals has drawn increasing attention in recent years due to the enormous potential for Brain-computer interfaces (BCIs). However, the remarkable discrepancy between the subject-dependent EEG representation and the semantic-dependent text representation poses a great challenge to this task. To mitigate this challenge, we devise a Curriculum Semantic-aware Contrastive Learning strategy (C-SCL), which effectively re-calibrates the subject-dependent EEG representation to the semantic-dependent EEG representation, thus reducing the discrepancy. Specifically, our C-SCL pulls semantically similar EEG representations together while pushing apart dissimilar ones. Besides, in order to introduce more meaningful contrastive pairs, we carefully employ curriculum learning to not only craft meaningful contrastive pairs but also make the learning progressively. We conduct extensive experiments on the ZuCo benchmark and our method combined with diverse models and architectures shows stable improvements across three types of metrics while achieving the new state-of-the-art. Further investigation proves not only its superiority in both the single-subject and low-resource settings but also its robust generalizability in the zero-shot setting.
Abstract (translated)
电泳脑电图生成(EEG-to-Text)旨在从EEG信号中直接生成自然文本,近年来因为脑机接口(BCI)的巨大潜力而引起了越来越多的关注。然而,主语依赖EEG表示和语义依赖文本表示之间的显著差异对该任务构成了一个巨大的挑战。为了减轻这个挑战,我们设计了一项 curriculum 语义 aware Contrastive Learning 策略(C-SCL),该策略有效地将主语依赖EEG表示转换为语义依赖EEG表示,从而减少了差异。具体来说,我们的 C-SCL 将语义相似的EEG表示聚合在一起,同时将不相似的表示分离。此外,为了引入更多有意义的对比对,我们 carefully运用了课程学习,不仅创建了有意义的对比对,而且逐步推进学习。我们在 ZuCo 基准测试中进行了大量实验,我们的方法与多种模型和架构组合在一起,表现出稳定的改进,同时实现了新的先进技术。进一步研究证明,不仅在单个主语和资源匮乏的环境下,该方法优越于其他方法,而且在零样本环境下,其鲁棒的泛化能力也非常出色。
URL
https://arxiv.org/abs/2301.09237