Abstract
Hand gestures are a primary output of the human motor system, yet the decoding of their neuromuscular signatures remains a bottleneck for basic neuroscience and assistive technologies such as prosthetics. Traditional human-machine interface pipelines rely on a single biosignal modality, but multimodal fusion can exploit complementary information from sensors. We systematically compare linear and attention-based fusion strategies across three architectures: a Multimodal MLP, a Multimodal Transformer, and a Hierarchical Transformer, evaluating performance on scenarios with unimodal and multimodal inputs. Experiments use two publicly available datasets: NinaPro DB2 (sEMG and accelerometer) and HD-sEMG 65-Gesture (high-density sEMG and force). Across both datasets, the Hierarchical Transformer with attention-based fusion consistently achieved the highest accuracy, surpassing the multimodal and best single-modality linear-fusion MLP baseline by over 10% on NinaPro DB2 and 3.7% on HD-sEMG. To investigate how modalities interact, we introduce an Isolation Network that selectively silences unimodal or cross-modal attention pathways, quantifying each group of token interactions' contribution to downstream decisions. Ablations reveal that cross-modal interactions contribute approximately 30% of the decision signal across transformer layers, highlighting the importance of attention-driven fusion in harnessing complementary modality information. Together, these findings reveal when and how multimodal fusion would enhance biosignal classification and also provides mechanistic insights of human muscle activities. The study would be beneficial in the design of sensor arrays for neurorobotic systems.
Abstract (translated)
手势是人类运动系统的主要输出之一，但对其神经肌肉特征的解码仍然是基础神经科学和假肢等辅助技术的一个瓶颈。传统的“人机接口”管道依赖单一类型的生物信号模式，而多模态融合可以通过利用来自不同传感器互补的信息来改进这一点。我们在三种架构（一个多模态MLP、一个多模态Transformer以及一个层级式Transformer）中系统地比较了线性与基于注意力的融合策略在单模态和多模态输入场景下的性能表现。实验使用两个公开数据集：NinaPro DB2（表面肌电信号sEMG和加速度计）和HD-sEMG 65-Gesture（高密度sEMG和力）。在这两个数据集中，基于注意力的融合层级式Transformer在所有情况下都达到了最高的准确度，在NinaPro DB2上超过了多模态和最佳单模态线性融合MLP基准超过10%，在HD-sEMG上则提高了3.7%。 为了探讨模式如何相互作用，我们引入了一个隔离网络，可以有选择地抑制单模态或跨模态注意力路径，并量化每组令牌互动对下游决策的贡献。消融实验显示，跨模态交互在整个Transformer层中为决策信号贡献了约30%，突显了基于注意机制融合在利用互补模式信息中的重要性。 这些发现揭示了多模态融合何时以及如何增强生物信号分类，并提供了有关人类肌肉活动的机理见解。该研究对于神经机器系统传感器阵列的设计具有指导意义。
URL
https://arxiv.org/abs/2506.16744
