Abstract
This study presents TSLFormer, a light and robust word-level Turkish Sign Language (TSL) recognition model that treats sign gestures as ordered, string-like language. Instead of using raw RGB or depth videos, our method only works with 3D joint positions - articulation points - extracted using Google's Mediapipe library, which focuses on the hand and torso skeletal locations. This creates efficient input dimensionality reduction while preserving important semantic gesture information. Our approach revisits sign language recognition as sequence-to-sequence translation, inspired by the linguistic nature of sign languages and the success of transformers in natural language processing. Since TSLFormer uses the self-attention mechanism, it effectively captures temporal co-occurrence within gesture sequences and highlights meaningful motion patterns as words unfold. Evaluated on the AUTSL dataset with over 36,000 samples and 227 different words, TSLFormer achieves competitive performance with minimal computational cost. These results show that joint-based input is sufficient for enabling real-time, mobile, and assistive communication systems for hearing-impaired individuals.
Abstract (translated)
本研究提出了TSLFormer，这是一种轻量且鲁棒的土耳其手语（TSL）单词级识别模型，将手势视为有序的语言字符串。与使用原始RGB或深度视频不同，我们的方法仅利用通过Google的Mediapipe库提取的手部和躯干骨骼位置的3D关节坐标作为输入。这在减少输入维度的同时保留了重要的语义手势信息。我们借鉴手语语言学性质及Transformer模型在自然语言处理中的成功经验，重新审视手语识别为序列到序列翻译问题。由于TSLFormer采用自注意力机制，它能够有效捕捉手势序列内的时间共现关系，并突出展示动作模式的含义。经过包含超过36,000个样本和227个不同单词的AUTSL数据集评估后，TSLFormer在计算成本最小化的情况下实现了竞争性的性能表现。这些结果表明基于关节输入足以支持听障人士所需的实时、便携及辅助通信系统。
URL
https://arxiv.org/abs/2505.07890
