Abstract
Sign language recognition (SLR) refers to interpreting sign language glosses from given videos automatically. This research area presents a complex challenge in computer vision because of the rapid and intricate movements inherent in sign languages, which encompass hand gestures, body postures, and even facial expressions. Recently, skeleton-based action recognition has attracted increasing attention due to its ability to handle variations in subjects and backgrounds independently. However, current skeleton-based SLR methods exhibit three limitations: 1) they often neglect the importance of realistic hand poses, where most studies train SLR models on non-realistic skeletal representations; 2) they tend to assume complete data availability in both training or inference phases, and capture intricate relationships among different body parts collectively; 3) these methods treat all sign glosses uniformly, failing to account for differences in complexity levels regarding skeletal representations. To enhance the realism of hand skeletal representations, we present a kinematic hand pose rectification method for enforcing constraints. Mitigating the impact of missing data, we propose a feature-isolated mechanism to focus on capturing local spatial-temporal context. This method captures the context concurrently and independently from individual features, thus enhancing the robustness of the SLR model. Additionally, to adapt to varying complexity levels of sign glosses, we develop an input-adaptive inference approach to optimise computational efficiency and accuracy. Experimental results demonstrate the effectiveness of our approach, as evidenced by achieving a new state-of-the-art (SOTA) performance on WLASL100 and LSA64. For WLASL100, we achieve a top-1 accuracy of 86.50\%, marking a relative improvement of 2.39% over the previous SOTA. For LSA64, we achieve a top-1 accuracy of 99.84%.
Abstract (translated)
手语识别（SLR）是指自动解析给定视频中的手势语言简写。这一研究领域在计算机视觉中构成了一个复杂挑战，原因在于手势语言包含了快速且复杂的动作，涉及手部姿势、身体姿态及面部表情等多方面因素。近年来，基于骨架的动作识别因其能够独立处理主体和背景的变化而吸引了越来越多的关注。然而，目前的基于骨架的手语识别方法存在三个局限性：1）它们通常忽视了真实手部姿势的重要性，在大多数研究中训练SLR模型使用的是非真实的骨骼表示；2）这些方法倾向于假设在训练或推断阶段数据是完整的，并且能够集体捕捉不同身体部位之间的复杂关系；3）这些方法对待所有的手势简写都采用统一处理的方式，未能考虑到关于骨架表示的复杂程度差异。 为了增强手部骨骼表示的真实性，我们提出了一种动力学手部姿势矫正方法，用于施加约束。为减轻数据缺失的影响，我们提出了一个特征隔离机制来专注于捕捉局部时空上下文，这种方法能够同时独立地从个别特征中捕获上下文信息，从而增强了SLR模型的鲁棒性。此外，为了适应手势简写复杂程度的不同，我们开发了一种输入自适应推断方法以优化计算效率和准确性。 实验结果表明了我们方法的有效性，具体表现为在WLASL100和LSA64数据集上实现了新的最先进（SOTA）性能表现。对于WLASL100数据集，我们的模型达到了86.50%的Top-1准确率，相较于之前的最先进水平提高了2.39%；而对于LSA64数据集，我们则实现了高达99.84%的Top-1准确率。
URL
https://arxiv.org/abs/2503.20436
