Abstract
Continuous sign language recognition (CSLR) requires precise spatio-temporal modeling to accurately recognize sequences of gestures in videos. Existing frameworks often rely on CNN-based spatial backbones combined with temporal convolution or recurrent modules. These techniques fail in capturing fine-grained hand and facial cues and modeling long-range temporal dependencies. To address these limitations, we propose the Unified Spatio-Temporal Modeling (USTM) framework, a spatio-temporal encoder that effectively models complex patterns using a combination of a Swin Transformer backbone enhanced with lightweight temporal adapter with positional embeddings (TAPE). Our framework captures fine-grained spatial features alongside short and long-term temporal context, enabling robust sign language recognition from RGB videos without relying on multi-stream inputs or auxiliary modalities. Extensive experiments on benchmarked datasets including PHOENIX14, PHOENIX14T, and CSL-Daily demonstrate that USTM achieves state-of-the-art performance against RGB-based as well as multi-modal CSLR approaches, while maintaining competitive performance against multi-stream approaches. These results highlight the strength and efficacy of the USTM framework for CSLR. The code is available at this https URL
Abstract (translated)
连续手语识别（CSLR）需要精确的时空建模，以准确地从视频中识别手势序列。现有的框架通常依赖于基于CNN的空间骨干网络，并结合时间卷积或递归模块。这些技术无法捕捉精细的手部和面部线索，也无法建立长时间的时序依赖关系。为了克服这些限制，我们提出了统一时空模型（USTM）框架，这是一种时空编码器，通过将Swin Transformer骨干网与轻量级的时间适配器结合使用，并添加位置嵌入（TAPE），来有效地建模复杂模式。我们的框架可以捕捉精细的视觉特征以及短期和长期时序上下文，从而能够仅从RGB视频中进行稳健的手语识别，而无需依赖多流输入或辅助模式。在包括PHOENIX14、PHOENIX14T和CSL-Daily在内的基准数据集上的广泛实验表明，USTM框架在基于RGB的方法以及多模态CSLR方法方面取得了最先进的性能，并且与多流方法相比保持了竞争性的表现。这些结果突显了USTM框架在CSLR中的强大有效性和适用性。代码可以在提供的链接中获取。
URL
https://arxiv.org/abs/2512.13415
