Abstract
Continuously recognizing sign gestures and converting them to glosses plays a key role in bridging the gap between the hearing and hearing-impaired communities. This involves recognizing and interpreting the hands, face, and body gestures of the signer, which pose a challenge as it involves a combination of all these features. Continuous Sign Language Recognition (CSLR) methods rely on multi-stage pipelines that first extract visual features, then align variable-length sequences with target glosses using CTC or HMM-based approaches. However, these alignment-based methods suffer from error propagation across stages, overfitting, and struggle with vocabulary scalability due to the intermediate gloss representation bottleneck. To address these limitations, we propose AutoSign, an autoregressive decoder-only transformer that directly translates pose sequences to natural language text, bypassing traditional alignment mechanisms entirely. The use of this decoder-only approach allows the model to directly map between the features and the glosses without the need for CTC loss while also directly learning the textual dependencies in the glosses. Our approach incorporates a temporal compression module using 1D CNNs to efficiently process pose sequences, followed by AraGPT2, a pre-trained Arabic decoder, to generate text (glosses). Through comprehensive ablation studies, we demonstrate that hand and body gestures provide the most discriminative features for signer-independent CSLR. By eliminating the multi-stage pipeline, AutoSign achieves substantial improvements on the Isharah-1000 dataset, achieving an improvement of up to 6.1\% in WER score compared to the best existing method.
Abstract (translated)
持续识别手势并将其转换为语符（glosses）在连接听觉正常人群和听力障碍人群之间发挥着关键作用。这涉及到识别和解释手语者的双手、面部及身体动作，由于这些特征的组合使用而带来了一定挑战。连续手语识别（Continuous Sign Language Recognition, CSLR）方法依赖于多阶段流水线技术，该技术首先提取视觉特征，然后通过CTC或HMM基方法将不同长度的时间序列与目标语符对齐。然而，基于这种方法存在跨阶段的错误传播、过拟合以及由于中间语符表示瓶颈而导致词汇量扩展困难的问题。 为了解决这些问题，我们提出了一种名为AutoSign的方法，这是一种自回归解码器专用的Transformer模型，它直接将姿态序列翻译成自然语言文本，完全绕过了传统的对齐机制。使用这种解码器专有方法使得该模型能够在不依赖CTC损失的情况下直接映射特征与语符，并且还可以直接学习语符中的文本依赖关系。我们的方法利用了一维CNN的时间压缩模块来高效处理姿态序列，随后采用预训练的阿拉伯语言解码器AraGPT2生成文本（语符）。通过全面的消融研究，我们证明了手部和身体动作提供了对说话人无关CSLR而言最具区分性的特征。通过消除多阶段管道，AutoSign在Isharah-1000数据集上实现了显著改进，与现有最佳方法相比，WER得分提高了高达6.1%。 简而言之，这项工作展示了使用自回归解码器专用Transformer模型进行直接姿态到文本翻译的潜力，并通过有效的时间压缩模块解决了连续手语识别中的关键问题。这种方法不仅简化了传统CSLR流程，还提升了系统的整体性能和灵活性。
URL
https://arxiv.org/abs/2507.19840
