Abstract
Online micro gesture recognition from hand skeletons is critical for VR/AR interaction but faces challenges due to limited public datasets and task-specific algorithms. Micro gestures involve subtle motion patterns, which make constructing datasets with precise skeletons and frame-level annotations difficult. To this end, we develop a multi-view self-supervised pipeline to automatically generate skeleton data, complemented by heuristic rules and expert refinement for semi-automatic annotation. Based on this pipeline, we introduce OMG-Bench, the first large-scale public benchmark for skeleton-based online micro gesture recognition. It features 40 fine-grained gesture classes with 13,948 instances across 1,272 sequences, characterized by subtle motions, rapid dynamics, and continuous execution. To tackle these challenges, we propose Hierarchical Memory-Augmented Transformer (HMATr), an end-to-end framework that unifies gesture detection and classification by leveraging hierarchical memory banks which store frame-level details and window-level semantics to preserve historical context. In addition, it employs learnable position-aware queries initialized from the memory to implicitly encode gesture positions and semantics. Experiments show that HMATr outperforms state-of-the-art methods by 7.6\% in detection rate, establishing a strong baseline for online micro gesture recognition. Project page: this https URL
Abstract (translated)
基于手骨骼的在线微手势识别对于虚拟现实（VR）和增强现实（AR）中的交互至关重要，但受限于公开数据集较少及任务特定算法的发展缓慢而面临挑战。由于微手势涉及细微的动作模式，构建具有精确骨骼信息和帧级注释的数据集变得困难。为此，我们开发了一种多视角自监督流水线来自动生成骨架数据，并辅以启发式规则和专家校准进行半自动注解。基于此流程，我们引入了OMG-Bench，这是第一个用于基于骨架的在线微手势识别的大规模公开基准测试平台。该平台包含40个精细的手势类别，共有13,948个实例及1,272条序列，这些实例和序列以细微动作、快速变化以及连续执行为特点。 为了应对上述挑战，我们提出了分层记忆增强变换器（Hierarchical Memory-Augmented Transformer, HMATr），这是一个统一手势检测与分类的端到端框架。该框架利用存储帧级细节及窗口级语义的信息层级存储库来保持历史上下文，并通过从内存中初始化的学习位置感知查询隐式编码手势的位置和语义。 实验表明，HMATr在检测率上比现有最佳方法高出7.6%，为在线微手势识别建立了一个强大的基准。项目页面：[此URL](https://this-url.com)
URL
https://arxiv.org/abs/2512.16727
