Abstract
Real-time Magnetic Resonance Imaging (rtMRI) visualizes vocal tract action, offering a comprehensive window into speech articulation. However, its signals are high dimensional and noisy, hindering interpretation. We investigate compact representations of spatiotemporal articulatory dynamics for phoneme recognition from midsagittal vocal tract rtMRI videos. We compare three feature types: (1) raw video, (2) optical flow, and (3) six linguistically-relevant regions of interest (ROIs) for articulator movements. We evaluate models trained independently on each representation, as well as multi-feature combinations. Results show that multi-feature models consistently outperform single-feature baselines, with the lowest phoneme error rate (PER) of 0.34 obtained by combining ROI and raw video. Temporal fidelity experiments demonstrate a reliance on fine-grained articulatory dynamics, while ROI ablation studies reveal strong contributions from tongue and lips. Our findings highlight how rtMRI-derived features provide accuracy and interpretability, and establish strategies for leveraging articulatory data in speech processing.
Abstract (translated)
实时磁共振成像（rtMRI）能够可视化声带动作，为语音发音提供了一个全面的窗口。然而，其信号具有高维度和噪音的特点，这阻碍了对其的理解。我们研究了用于从中矢状面声带rtMRI视频中识别音素的时空构音动力学的紧凑表示形式。我们将三种类型的特征进行了比较：（1）原始视频、（2）光学流以及（3）六个与语言相关的感兴趣区域（ROI），这些区域代表发音器的动作。我们评估了在每种表示类型上单独训练的模型，以及多特征组合的效果。 实验结果表明，多特征模型始终优于单一特征基线，在将ROI和原始视频结合后实现了最低的音素错误率（PER）0.34。时间保真度实验展示了对精细构音动态的依赖性，而ROI消融研究揭示了舌头和嘴唇的贡献尤为显著。 我们的研究成果突显了从rtMRI衍生特征中获得准确性与可解释性的潜力，并为在语音处理中利用构音数据建立了策略。
URL
https://arxiv.org/abs/2509.15689
