Abstract
Self-supervised learning (SSL) has shown remarkable success in skeleton-based action recognition by leveraging data augmentations to learn meaningful representations. However, existing SSL methods rely on data augmentations that predominantly focus on masking high-motion frames and high-degree joints such as joints with degree 3 or 4. This results in biased and incomplete feature representations that struggle to generalize across varied motion patterns. To address this, we propose Asymmetric Spatio-temporal Masking (ASMa) for Skeleton Action Representation Learning, a novel combination of masking to learn a full spectrum of spatio-temporal dynamics inherent in human actions. ASMa employs two complementary masking strategies: one that selectively masks high-degree joints and low-motion, and another that masks low-degree joints and high-motion frames. These masking strategies ensure a more balanced and comprehensive skeleton representation learning. Furthermore, we introduce a learnable feature alignment module to effectively align the representations learned from both masked views. To facilitate deployment in resource-constrained settings and on low-resource devices, we compress the learned and aligned representation into a lightweight model using knowledge distillation. Extensive experiments on NTU RGB+D 60, NTU RGB+D 120, and PKU-MMD datasets demonstrate that our approach outperforms existing SSL methods with an average improvement of 2.7-4.4% in fine-tuning and up to 5.9% in transfer learning to noisy datasets and achieves competitive performance compared to fully supervised baselines. Our distilled model achieves 91.4% parameter reduction and 3x faster inference on edge devices while maintaining competitive accuracy, enabling practical deployment in resource-constrained scenarios.
Abstract (translated)
自监督学习（SSL）在基于骨架的动作识别中取得了显著的成功，通过利用数据增强来学习有意义的表示。然而，现有的SSL方法依赖于主要关注遮蔽高运动帧和高阶关节（如3度或4度关节）的数据增强，这导致了有偏且不完整的特征表示，难以泛化到各种运动模式上。为了解决这个问题，我们提出了不对称时空遮蔽（ASMa），这是一种用于骨架动作表征学习的新颖组合方法，它通过一种新的掩码策略来全面学习人体动作中内在的时空动态特性。 ASMa采用两种互补的掩码策略：一是选择性地对高阶关节和低运动进行掩码处理；二是对低阶关节和高运动帧进行掩码处理。这两种掩码策略确保了更平衡和综合的骨架表示学习过程。此外，我们引入了一个可学习的功能对齐模块，以有效地将从两种被遮蔽视图中学到的表征进行对齐。 为了便于在资源受限环境及低资源设备上部署，我们将学到并对齐后的表征压缩成一个轻量级模型，并使用知识蒸馏技术来实现这一点。在NTU RGB+D 60、NTU RGB+D 120和PKU-MMD数据集上的广泛实验表明，我们的方法优于现有的SSL方法，在微调阶段平均提高了2.7-4.4%，并在迁移到嘈杂的数据集时最多提升了5.9%。此外，在与完全监督的基线相比时，我们的方法也达到了竞争性的性能。 压缩后的模型实现了高达91.4%的参数减少，并且在边缘设备上推理速度提高了3倍，同时保持了竞争力的准确性，这使得在资源受限场景下的实际部署成为可能。
URL
https://arxiv.org/abs/2602.06251
