Abstract
Mid-air gestures in Extended Reality (XR) often cause fatigue and imprecision. Surface-based interactions offer improved accuracy and comfort, but current egocentric vision methods struggle due to hand tracking challenges and unreliable surface plane estimation. We introduce SurfaceXR, a sensor fusion approach combining headset-based hand tracking with smartwatch IMU data to enable robust inputs on everyday surfaces. Our insight is that these modalities are complementary: hand tracking provides 3D positional data while IMUs capture high-frequency motion. A 21-participant study validates SurfaceXR's effectiveness for touch tracking and 8-class gesture recognition, demonstrating significant improvements over single-modality approaches.
Abstract (translated)
在扩展现实（XR）中，空中手势常导致疲劳和精度不足。基于表面的交互虽能提升准确性与舒适度，但当前第一人称视觉方法受限于手部追踪困难与不可靠的表面平面估计。我们提出SurfaceXR，一种通过头显手部追踪与智能手表惯性测量单元数据融合的方案，可在日常表面实现稳健输入。我们的核心见解在于这些模态具有互补性：手部追踪提供三维位置数据，而惯性测量单元能捕捉高频运动。一项涵盖21名参与者的研究验证了SurfaceXR在触摸追踪与八类手势识别中的有效性，相较于单模态方法展现出显著改进。
URL
https://arxiv.org/abs/2603.19529
