Abstract
Recent advances in sign language research have benefited from CNN-based backbones, which are primarily transferred from traditional computer vision tasks (\eg object identification, image recognition). However, these CNN-based backbones usually excel at extracting features like contours and texture, but may struggle with capturing sign-related features. In fact, sign language tasks require focusing on sign-related regions, including the collaboration between different regions (\eg left hand region and right hand region) and the effective content in a single region. To capture such region-related features, we introduce MixSignGraph, which represents sign sequences as a group of mixed graphs and designs the following three graph modules for feature extraction, \ie Local Sign Graph (LSG) module, Temporal Sign Graph (TSG) module and Hierarchical Sign Graph (HSG) module. Specifically, the LSG module learns the correlation of intra-frame cross-region features within one frame, \ie focusing on spatial features. The TSG module tracks the interaction of inter-frame cross-region features among adjacent frames, \ie focusing on temporal features. The HSG module aggregates the same-region features from different-granularity feature maps of a frame, \ie focusing on hierarchical features. In addition, to further improve the performance of sign language tasks without gloss annotations, we propose a simple yet counter-intuitive Text-driven CTC Pre-training (TCP) method, which generates pseudo gloss labels from text labels for model pre-training. Extensive experiments conducted on current five public sign language datasets demonstrate the superior performance of the proposed model. Notably, our model surpasses the SOTA models on multiple sign language tasks across several datasets, without relying on any additional cues.
Abstract (translated)
最近的手语研究进展得益于基于卷积神经网络（CNN）的骨干架构，这些架构主要从传统的计算机视觉任务中转移而来（例如物体识别、图像识别）。然而，这类基于CNN的骨干通常擅长提取轮廓和纹理等特征，但可能难以捕捉与手语相关的特征。实际上，手语任务需要关注特定的手语相关区域，包括不同区域之间的协作（如左手区域和右手区域）以及单个区域内有效内容的理解。为了捕获这些区域相关的特征，我们引入了MixSignGraph模型，该模型将手势序列表示为混合图组，并设计了三个用于特征提取的图模块：即局部手语图（LSG）模块、时间手语图（TSG）模块和层次化手语图（HSG）模块。具体来说，LSG模块学习同一帧内跨区域特征之间的相关性，即关注空间特征；TSG模块跟踪相邻帧间跨区域特征的交互作用，即关注时间特征；而HSG模块聚合来自单个帧不同粒度特征图中的相同区域特征，即关注层次化特征。此外，为了进一步提高无音节注释的手语任务性能，我们提出了一种简单却反直觉的文本驱动CTC预训练（TCP）方法，该方法从文本标签中生成伪音节标签进行模型预训练。在目前五个公开的手语数据集上进行的广泛实验表明了所提模型的优越性。值得注意的是，在多个手语任务和几个数据集中，我们的模型无需任何额外线索即可超越最先进的模型。
URL
https://arxiv.org/abs/2504.12020
