Abstract
Restoring speech communication from neural signals is a central goal of brain-computer interface research, yet EEG-based speech reconstruction remains challenging due to limited spatial resolution, susceptibility to noise, and the absence of temporally aligned acoustic targets in imagined speech. In this study, we propose an EEG-to-Voice paradigm that directly reconstructs speech from non-invasive EEG signals without dynamic time warping (DTW) or explicit temporal alignment. The proposed pipeline generates mel-spectrograms from EEG in an open-loop manner using a subject-specific generator, followed by pretrained vocoder and automatic speech recognition (ASR) modules to synthesize speech waveforms and decode text. Separate generators were trained for spoken speech and imagined speech, and transfer learning-based domain adaptation was applied by pretraining on spoken speech and adapting to imagined speech. A minimal language model-based correction module was optionally applied to correct limited ASR errors while preserving semantic structure. The framework was evaluated under 2 s and 4 s speech conditions using acoustic-level metrics (PCC, RMSE, MCD) and linguistic-level metrics (CER, WER). Stable acoustic reconstruction and comparable linguistic accuracy were observed for both spoken speech and imagined speech. While acoustic similarity decreased for longer utterances, text-level decoding performance was largely preserved, and word-position analysis revealed a mild increase in decoding errors toward later parts of sentences. The language model-based correction consistently reduced CER and WER without introducing semantic distortion. These results demonstrate the feasibility of direct, open-loop EEG-to-Voice reconstruction for spoken speech and imagined speech without explicit temporal alignment.
Abstract (translated)
从神经信号恢复语音通信是脑机接口研究的主要目标之一，但基于EEG的语音重构由于空间分辨率有限、易受噪声影响以及缺乏想象语音中与时间同步的声音目标而仍具挑战性。本研究提出了一种直接从非侵入式EEG信号重建语音的EEG-to-Voice范式，并且在该过程中不使用动态时间规整（DTW）或显式的时序对齐方法。 此框架包括一个基于个体生成器的闭环流程，该生成器能够从EEG信号中生成梅尔频谱图，之后再经过预训练的声音合成器和自动语音识别(ASR)模块以合成交互式波形并进行解码。针对口语和想象中的言语分别训练了不同的生成器，并通过先在口语上进行预训练然后适应于想象中的言语的方法应用了基于迁移学习的领域自适应技术。可选地，应用了一个基于最小语言模型的校正模块来修正有限的ASR错误同时保持语义结构。 该框架针对2秒和4秒语音条件进行了评估，并使用声学级（PCC、RMSE、MCD）和语言级指标（CER、WER）进行评价。结果显示，无论是口语还是想象中的言语，在声音重建稳定性和语言准确性方面都表现出一致的结果。尽管在较长的话语中声学相似性有所下降，但文本级别的解码性能大体上得以保持，并且通过对单词位置的分析发现，句子后部部分的解码错误略有增加。基于语言模型的校正模块持续降低了CER和WER，而不会引入语义扭曲。 这些结果展示了直接、闭环的EEG-to-Voice重构在不使用显式时序对齐的情况下实现口语和想象中的言语的可能性。
URL
https://arxiv.org/abs/2512.22146
