Abstract
Continuous sign language recognition (CSLR) aims to transcribe untrimmed videos into glosses, which are typically textual words. Recent studies indicate that the lack of large datasets and precise annotations has become a bottleneck for CSLR due to insufficient training data. To address this, some works have developed cross-modal solutions to align visual and textual modalities. However, they typically extract textual features from glosses without fully utilizing their knowledge. In this paper, we propose the Hierarchical Sub-action Tree (HST), termed HST-CSLR, to efficiently combine gloss knowledge with visual representation learning. By incorporating gloss-specific knowledge from large language models, our approach leverages textual information more effectively. Specifically, we construct an HST for textual information representation, aligning visual and textual modalities step-by-step and benefiting from the tree structure to reduce computational complexity. Additionally, we impose a contrastive alignment enhancement to bridge the gap between the two modalities. Experiments on four datasets (PHOENIX-2014, PHOENIX-2014T, CSL-Daily, and Sign Language Gesture) demonstrate the effectiveness of our HST-CSLR.
Abstract (translated)
连续手语识别（CSLR）旨在将未修剪的视频转换为文字形式的手语词汇，即“gloss”。最近的研究表明，由于缺乏大规模数据集和精准标注，CSLR的发展受到了训练数据不足的瓶颈限制。为了应对这一挑战，一些研究工作开发了跨模态解决方案来对齐视觉与文本模态。然而，这些方法通常只从gloss中提取文本特征，而没有充分利用其知识。 本文提出了一种层次子动作树（Hierarchical Sub-action Tree, HST），命名为HST-CSLR，旨在高效地结合gloss知识和视觉表示学习。通过整合大型语言模型中的手语特定知识，我们的方法更有效地利用了文字信息。具体而言，我们构建了一个用于文本信息表达的HST，逐步对齐视觉与文本模态，并从树结构中受益以降低计算复杂性。此外，我们引入了一种对比式对齐增强机制来弥合两个模态之间的差距。 在四个数据集（PHOENIX-2014、PHOENIX-2014T、CSL-Daily和Sign Language Gesture）上的实验表明了HST-CSLR的有效性。
URL
https://arxiv.org/abs/2506.20947
