Abstract
Human operators are still frequently exposed to hazardous environments such as disaster zones and industrial facilities, where intuitive and reliable teleoperation of mobile robots and Unmanned Aerial Vehicles (UAVs) is essential. In this context, hands-free teleoperation enhances operator mobility and situational awareness, thereby improving safety in hazardous environments. While vision-based gesture recognition has been explored as one method for hands-free teleoperation, its performance often deteriorates under occlusions, lighting variations, and cluttered backgrounds, limiting its applicability in real-world operations. To overcome these limitations, we propose a multimodal gesture recognition framework that integrates inertial data (accelerometer, gyroscope, and orientation) from Apple Watches on both wrists with capacitive sensing signals from custom gloves. We design a late fusion strategy based on the log-likelihood ratio (LLR), which not only enhances recognition performance but also provides interpretability by quantifying modality-specific contributions. To support this research, we introduce a new dataset of 20 distinct gestures inspired by aircraft marshalling signals, comprising synchronized RGB video, IMU, and capacitive sensor data. Experimental results demonstrate that our framework achieves performance comparable to a state-of-the-art vision-based baseline while significantly reducing computational cost, model size, and training time, making it well suited for real-time robot control. We therefore underscore the potential of sensor-based multimodal fusion as a robust and interpretable solution for gesture-driven mobile robot and drone teleoperation.
Abstract (translated)
人类操作员仍然经常暴露在危险环境中，如灾区和工业设施中，在这些地方，直观且可靠的移动机器人和无人驾驶飞行器（UAV）远程操控是至关重要的。在这种情况下，无手操控制（hands-free teleoperation）能够增强操作人员的机动性和情境意识，从而提高危险环境中的安全性。虽然基于视觉的手势识别已被探索为一种无手操控的方法，但由于遮挡、光照变化和杂乱背景的影响，其性能经常下降，限制了其实用性。 为了克服这些局限性，我们提出了一种多模态手势识别框架，该框架集成了来自双腕Apple Watch的惯性数据（加速度计、陀螺仪和方向）以及定制手套提供的电容感应信号。我们设计了一种基于对数似然比（LLR）的晚期融合策略，这种策略不仅提升了识别性能，还通过量化各模式的具体贡献提供了可解释性。 为了支持这项研究，我们引入了一个新的数据集，该数据集中包含20个不同的手势，这些手势源自飞机引导信号。这个新数据集包括同步的RGB视频、IMU和电容传感器数据。实验结果表明，我们的框架在性能上与最先进的基于视觉的方法相媲美，同时显著降低了计算成本、模型大小和训练时间，使其非常适合实时机器人控制。 因此，我们强调传感器驱动多模态融合作为手势驱动移动机器人和无人机远程操控的稳健且可解释解决方案的巨大潜力。
URL
https://arxiv.org/abs/2602.23694
