Abstract
The absence of effective communication the deaf population represents the main social gap in this community. Furthermore, the sign language, main deaf communication tool, is unlettered, i.e., there is no formal written representation. In consequence, main challenge today is the automatic translation among spatiotemporal sign representation and natural text language. Recent approaches are based on encoder-decoder architectures, where the most relevant strategies integrate attention modules to enhance non-linear correspondences, besides, many of these approximations require complex training and architectural schemes to achieve reasonable predictions, because of the absence of intermediate text projections. However, they are still limited by the redundant background information of the video sequences. This work introduces a multitask transformer architecture that includes a gloss learning representation to achieve a more suitable translation. The proposed approach also includes a dense motion representation that enhances gestures and includes kinematic information, a key component in sign language. From this representation it is possible to avoid background information and exploit the geometry of the signs, in addition, it includes spatiotemporal representations that facilitate the alignment between gestures and glosses as an intermediate textual representation. The proposed approach outperforms the state-of-the-art evaluated on the CoL-SLTD dataset, achieving a BLEU-4 of 72,64% in split 1, and a BLEU-4 of 14,64% in split 2. Additionally, the strategy was validated on the RWTH-PHOENIX-Weather 2014 T dataset, achieving a competitive BLEU-4 of 11,58%.
Abstract (translated)
聋人群体中有效沟通的缺乏代表了这一社区的主要社会差距。此外，手语作为主要的交流工具是没有文字形式的，即没有正式的文字表示方式。因此，当前的主要挑战在于在空间时间手势表示与自然语言文本之间实现自动翻译。 最近的方法基于编码器-解码器架构，其中最相关的策略是集成注意力模块来增强非线性对应关系。然而，由于缺乏中间文本投影，许多这些方法需要复杂的训练和体系结构方案才能获得合理的预测结果。尽管如此，它们仍然受到视频序列中冗余背景信息的限制。 这项工作介绍了一种多任务变压器架构，该架构包括手势词学习表示以实现更合适的翻译。所提出的方法还包括密集运动表示，增强了手势，并包含了关键点动力学信息，在手语中这是一个关键组成部分。从这种表示法中可以避免背景信息并利用手势的几何形状，此外它还包含空间时间表示形式，有助于在手势和手势词之间进行对齐作为中间文本表示。 所提出的方法在CoL-SLTD数据集上进行了评估，并超越了现有技术，在split 1中实现了72.64%的BLEU-4评分，在split 2中实现了14.64%的BLEU-4评分。此外，该策略还在RWTH-PHOENIX-Weather 2014 T数据集上进行了验证，并取得了具有竞争力的11.58%的BLEU-4得分。
URL
https://arxiv.org/abs/2503.19668
