Abstract
Early identification of stroke symptoms is essential for enabling timely intervention and improving patient outcomes, particularly in prehospital settings. This study presents a fast, non-invasive multimodal deep learning framework for automatic binary stroke screening based on data collected during the F.A.S.T. assessment. The proposed approach integrates complementary information from facial expressions, speech signals, and upper-body movements to enhance diagnostic robustness. Facial dynamics are represented using landmark based features and modeled with a Transformer architecture to capture temporal dependencies. Speech signals are converted into mel spectrograms and processed using an Audio Spectrogram Transformer, while upper-body pose sequences are analyzed with an MLP-Mixer network to model spatiotemporal motion patterns. The extracted modality specific representations are combined through an attention-based fusion mechanism to effectively learn cross modal interactions. Experiments conducted on a self-collected dataset of 222 videos from 37 subjects demonstrate that the proposed multimodal model consistently outperforms unimodal baselines, achieving 95.83% accuracy and a 96.00% F1-score. The model attains a strong balance between sensitivity and specificity and successfully detects all stroke cases in the test set. These results highlight the potential of multimodal learning and transfer learning for early stroke screening, while emphasizing the need for larger, clinically representative datasets to support reliable real-world deployment.
Abstract (translated)
早期识别中风症状对于及时干预和改善患者预后至关重要，尤其是在院前环境中。本研究提出了一种基于F.A.S.T评估期间收集的数据的快速、非侵入性多模态深度学习框架，用于自动二元中风筛查。该方法整合了面部表情、语音信号及上半身动作之间的互补信息，以增强诊断稳健性。 面部动态通过关键点特征表示，并采用Transformer架构建模，捕捉时间依赖关系；语音信号被转换为梅尔频谱图并使用音频频谱图Transformer进行处理；上半身姿态序列则利用MLP-Mixer网络分析空间-时间运动模式。各模态特异性的表征是通过基于注意力的融合机制结合在一起，以有效学习跨模态交互。 在自行收集的数据集（包含37名受试者222段视频）上的实验表明，所提出的多模态模型始终优于单一模态基准模型，在准确性方面达到95.83%，F1得分高达96%。该模型实现了高敏感性和特异性之间的良好平衡，并成功检测了测试集中所有中风案例。 这些结果突显了多模态学习和迁移学习在早期中风筛查中的潜力，同时也强调需要更大的、具有临床代表性的数据集来支持可靠的现实世界部署。
URL
https://arxiv.org/abs/2601.11896
