Abstract
Accurate recognition of sign language in healthcare communication poses a significant challenge, requiring frameworks that can accurately interpret complex multimodal gestures. To deal with this, we propose FusionEnsemble-Net, a novel attention-based ensemble of spatiotemporal networks that dynamically fuses visual and motion data to enhance recognition accuracy. The proposed approach processes RGB video and range Doppler map radar modalities synchronously through four different spatiotemporal networks. For each network, features from both modalities are continuously fused using an attention-based fusion module before being fed into an ensemble of classifiers. Finally, the outputs of these four different fused channels are combined in an ensemble classification head, thereby enhancing the model's robustness. Experiments demonstrate that FusionEnsemble-Net outperforms state-of-the-art approaches with a test accuracy of 99.44% on the large-scale MultiMeDaLIS dataset for Italian Sign Language. Our findings indicate that an ensemble of diverse spatiotemporal networks, unified by attention-based fusion, yields a robust and accurate framework for complex, multimodal isolated gesture recognition tasks. The source code is available at: this https URL.
Abstract (translated)
在医疗通信中准确识别手语是一项重大挑战，需要能够精确解析复杂多模态手势的框架。为应对这一问题，我们提出了FusionEnsemble-Net，这是一种基于注意力机制的时空网络集成模型，该模型可以动态融合视觉和运动数据以提高识别精度。提出的这种方法通过四种不同的时空网络同步处理RGB视频和范围多普勒地图雷达模式。对于每个网络，两种模态的数据特征连续地使用基于注意的融合模块进行融合后输入分类器集成。最后，这四个不同融合通道的输出在集成分类头中结合在一起，从而增强模型的鲁棒性。实验表明，在处理意大利手语的大规模MultiMeDaLIS数据集时，FusionEnsemble-Net以99.44%的测试准确率超过了当前最先进的方法。我们的研究结果表明，通过基于注意机制的融合统一起来的一系列时空网络集成提供了一种稳健且准确的方法来解决复杂多模态孤立手势识别任务。源代码可在以下网址获取：[此处插入链接]。
URL
https://arxiv.org/abs/2508.09362
