Abstract
Micro-gesture recognition (MGR) targets the identification of subtle and fine-grained human motions and requires accurate modeling of both long-range and local spatiotemporal dependencies. While CNNs are effective at capturing local patterns, they struggle with long-range dependencies due to their limited receptive fields. Transformer-based models address this limitation through self-attention mechanisms but suffer from high computational costs. Recently, Mamba has shown promise as an efficient model, leveraging state space models (SSMs) to enable linear-time processing However, directly applying the vanilla Mamba to MGR may not be optimal. This is because Mamba processes inputs as 1D sequences, with state updates relying solely on the previous state, and thus lacks the ability to model local spatiotemporal dependencies. In addition, previous methods lack a design of motion-awareness, which is crucial in MGR. To overcome these limitations, we propose motion-aware state fusion mamba (MSF-Mamba), which enhances Mamba with local spatiotemporal modeling by fusing local contextual neighboring states. Our design introduces a motion-aware state fusion module based on central frame difference (CFD). Furthermore, a multiscale version named MSF-Mamba+ has been proposed. Specifically, MSF-Mamba supports multiscale motion-aware state fusion, as well as an adaptive scale weighting module that dynamically weighs the fused states across different scales. These enhancements explicitly address the limitations of vanilla Mamba by enabling motion-aware local spatiotemporal modeling, allowing MSF-Mamba and MSF-Mamba to effectively capture subtle motion cues for MGR. Experiments on two public MGR datasets demonstrate that even the lightweight version, namely, MSF-Mamba, achieves SoTA performance, outperforming existing CNN-, Transformer-, and SSM-based models while maintaining high efficiency.
Abstract (translated)
微手势识别（Micro-gesture Recognition，MGR）的目标是识别细微且精细的人体动作，并需要准确建模长距离和局部的时空依赖关系。尽管卷积神经网络（CNNs）在捕捉局部模式方面表现出色，但由于其有限的感受野，它们难以处理长程依赖性。基于Transformer的模型通过自注意力机制解决了这一局限性，但面临着高计算成本的问题。最近，Mamba作为一种高效模型显示出潜力，它利用状态空间模型（State Space Models, SSMs）实现线性时间处理。然而，直接将标准的Mamba应用于微手势识别可能并非最佳选择，因为Mamba以一维序列方式处理输入，并且状态更新仅依赖于前一个状态，因此无法建模局部时空依赖关系。此外，先前的方法缺乏运动感知设计，在微手势识别中这一点至关重要。 为克服这些限制，我们提出了一种基于运动感知的状态融合mamba（Motion-aware State Fusion mamba, MSF-Mamba），通过融合本地上下文邻近状态来增强Mamba的局部时空建模能力。我们的设计方案引入了一个基于中央帧差（Central Frame Difference, CFD）的运动感知状态融合模块。此外，还提出了一种多尺度版本MSF-Mamba+，它支持多尺度的运动感知状态融合以及一个自适应缩放权重模块，在不同尺度之间动态加权融合后的状态。 这些改进明确解决了标准Mamba模型的局限性，通过启用基于运动的状态融合实现了局部时空建模。这使得MSF-Mamba及其扩展版本能够有效捕捉细微的动作线索，用于微手势识别任务中。在两个公开的微手势数据集上的实验表明，即便是轻量级版本的MSF-Mamba也达到了最先进的性能水平，并且优于现有的CNN、Transformer和SSM模型，同时保持了高效率。 总结来说，所提出的MSF-Mamba及其多尺度版本通过引入运动感知机制显著提高了在长程与局部时空依赖建模中的有效性，展示了微手势识别任务中卓越的性能与效率。
URL
https://arxiv.org/abs/2510.10478
