Abstract
Continuous sign language recognition (CSLR) focuses on interpreting and transcribing sequences of sign language gestures in videos. In this work, we propose CLIP sign language adaptation (CLIP-SLA), a novel CSLR framework that leverages the powerful pre-trained visual encoder from the CLIP model to sign language tasks through parameter-efficient fine-tuning (PEFT). We introduce two variants, SLA-Adapter and SLA-LoRA, which integrate PEFT modules into the CLIP visual encoder, enabling fine-tuning with minimal trainable parameters. The effectiveness of the proposed frameworks is validated on four datasets: Phoenix2014, Phoenix2014-T, CSL-Daily, and Isharah-500, where both CLIP-SLA variants outperformed several SOTA models with fewer trainable parameters. Extensive ablation studies emphasize the effectiveness and flexibility of the proposed methods with different vision-language models for CSLR. These findings showcase the potential of adapting large-scale pre-trained models for scalable and efficient CSLR, which pave the way for future advancements in sign language understanding.
Abstract (translated)
连续手语识别（CSLR）专注于解读和转录视频中的一系列手语手势。在这项工作中，我们提出了一种新颖的CSLR框架——CLIP手语适应（CLIP-SLA），该框架利用了从CLIP模型中预训练的强大视觉编码器，并通过参数高效微调（PEFT）将其应用于手语任务。我们引入了两种变体：SLA-Adapter和SLA-LoRA，它们将PEFT模块整合到CLIP视觉编码器中，从而可以在保持极少量可训练参数的情况下进行精细调整。 该框架的有效性在四个数据集上进行了验证：Phoenix2014、Phoenix2014-T、CSL-Daily 和 Isharah-500，在这些数据集中，两个 CLIP-SLA 变体均以更少的可训练参数优于多个最先进（SOTA）模型。广泛的消融研究表明了所提出的方法在不同的视觉语言模型中用于 CSLR 时的有效性和灵活性。 这些发现展示了大规模预训练模型适应于具有规模和效率优势的手语识别系统的潜力，为未来手语理解领域的进一步发展铺平了道路。
URL
https://arxiv.org/abs/2504.01666
