Abstract
Sign Language Recognition (SLR) systems primarily focus on manual gestures, but non-manual features such as mouth movements, specifically mouthing, provide valuable linguistic information. This work directly classifies mouthing instances to their corresponding words in the spoken language while exploring the potential of transfer learning from Visual Speech Recognition (VSR) to mouthing recognition in German Sign Language. We leverage three VSR datasets: one in English, one in German with unrelated words and one in German containing the same target words as the mouthing dataset, to investigate the impact of task similarity in this setting. Our results demonstrate that multi-task learning improves both mouthing recognition and VSR accuracy as well as model robustness, suggesting that mouthing recognition should be treated as a distinct but related task to VSR. This research contributes to the field of SLR by proposing knowledge transfer from VSR to SLR datasets with limited mouthing annotations.
Abstract (translated)
手语识别（SLR）系统主要关注手动手势，但非手动特征如嘴部动作——尤其是口型——提供了有价值的语言信息。本研究直接将口型实例分类到对应的口语词汇中，并探索从视觉语音识别（VSR）向德国手语中的口型识别转移学习的潜力。我们利用三个VSR数据集：一个英文的，一个德文但包含不相关的词语，以及另一个德文且包含与口型数据集相同目标词的数据集，来研究该设置下的任务相似性影响。我们的结果表明，在这种情况下，多任务学习可以提高口型识别和VSR的准确性以及模型鲁棒性，这表明应该将口型识别视为与VSR相关但独立的任务。这项研究通过从VSR向具有有限口型注释的手语数据集进行知识转移，为SLR领域做出了贡献。
URL
https://arxiv.org/abs/2505.13784
