Abstract
Sign Language Machine Translation (SLMT) aims to bridge communication between Deaf and hearing individuals. However, its progress is constrained by scarce datasets, limited signer diversity, and large domain gaps between sign motion patterns and pretrained representations. Existing transfer learning approaches in SLMT are static and often lead to overfitting. These challenges call for the development of an adaptive framework that preserves pretrained structure while remaining robust across linguistic and signing variations. To fill this void, we propose a Hierarchical Adaptive Transfer Learning (HATL) framework, where pretrained layers are progressively and dynamically unfrozen based on training performance behavior. HATL combines dynamic unfreezing, layer-wise learning rate decay, and stability mechanisms to preserve generic representations while adapting to sign characteristics. We evaluate HATL on Sign2Text and Sign2Gloss2Text translation tasks using a pretrained ST-GCN++ backbone for feature extraction and the Transformer and an adaptive transformer (ADAT)for translation. To ensure robust multilingual generalization, we evaluate the proposed approach across three datasets: RWTH-PHOENIXWeather-2014 (PHOENIX14T), Isharah, and MedASL. Experimental results show that HATL consistently outperforms traditional transfer learning approaches across tasks and models, with ADAT achieving BLEU-4 improvements of 15.0% on PHOENIX14T and Isharah and 37.6% on MedASL.
Abstract (translated)
手语机器翻译旨在促进听障人士与健听人士之间的沟通。然而，其发展受限于数据集稀缺、签署者多样性不足，以及手语动作模式与预训练表示之间存在较大领域差异。现有SLMT中的迁移学习方法通常是静态的，容易导致过拟合。这些挑战要求开发一种自适应框架，在保持预训练结构的同时，对语言和签署变异具有鲁棒性。为填补这一空白，我们提出了分层自适应迁移学习框架，该框架根据训练表现行为逐步动态解冻预训练层。HATL结合了动态解冻、逐层学习率衰减和稳定性机制，在适应手语特征的同时保留通用表示。我们使用预训练的ST-GCN++骨干网络进行特征提取，并采用Transformer及自适应Transformer进行翻译，在Sign2Text和Sign2Gloss2Text翻译任务上评估HATL。为确保稳健的多语言泛化能力，我们在三个数据集上评估该方法：RWTH-PHOENIX-Weather-2014（PHOENIX14T）、Isharah和MedASL。实验结果表明，HATL在各项任务和模型中均持续优于传统迁移学习方法，其中ADAT在PHOENIX14T和Isharah上实现了15.0%的BLEU-4提升，在MedASL上实现了37.6%的提升。
URL
https://arxiv.org/abs/2603.19260
