Abstract
Helping deaf and hard-of-hearing people communicate more easily is the main goal of Automatic Sign Language Translation. Although most past research has focused on turning sign language into text, doing the reverse, turning spoken English into sign language animations, has been largely overlooked. That's because it involves multiple steps, such as understanding speech, translating it into sign-friendly grammar, and generating natural human motion. In this work, we introduce a complete pipeline that converts English speech into smooth, realistic 3D sign language animations. Our system starts with Whisper to translate spoken English into text. Then, we use a MarianMT machine translation model to translate that text into American Sign Language (ASL) gloss, a simplified version of sign language that captures meaning without grammar. This model performs well, reaching BLEU scores of 0.7714 and 0.8923. To make the gloss translation more accurate, we also use word embeddings such as Word2Vec and FastText to understand word meanings. Finally, we animate the translated gloss using a 3D keypoint-based motion system trained on Sign3D-WLASL, a dataset we created by extracting body, hand, and face key points from real ASL videos in the WLASL dataset. To support the gloss translation stage, we also built a new dataset called BookGlossCorpus-CG, which turns everyday English sentences from the BookCorpus dataset into ASL gloss using grammar rules. Our system stitches everything together by smoothly interpolating between signs to create natural, continuous animations. Unlike previous works like How2Sign and Phoenix-2014T that focus on recognition or use only one type of data, our pipeline brings together audio, text, and motion in a single framework that goes all the way from spoken English to lifelike 3D sign language animation.
Abstract (translated)
帮助聋人和听力障碍者更轻松地交流是自动手语翻译的主要目标。尽管过去的研究大多集中在将手语转换为文本上，但将其反向操作即把口语英语转换成手语动画的工作却被很大程度上忽视了。这是因为后者涉及多个步骤，包括理解语音、将语音翻译成交互性强的手语语法，并生成自然的人体动作。在这一工作中，我们引入了一个完整的流水线，用于将英语口述语言转换为流畅且逼真的3D手语动画。 我们的系统首先使用Whisper将口语英语转换成文本。然后，采用MarianMT机器翻译模型将该文本翻译成美国手语（ASL）符号，这是一种简化了的手语形式，仅捕捉到意义而忽略了语法细节。这一模型表现良好，在BLEU分数上达到了0.7714和0.8923的评分。为了使符号翻译更加准确，我们还使用如Word2Vec和FastText这样的词嵌入技术来理解单词的意义。 最后，我们采用基于3D关键点的动作系统将翻译后的符号进行动画化处理，该动作系统在Sign3D-WLASL上进行了训练——这是一个我们从WLASL数据集中提取真实ASL视频中的身体、手部和面部关键点所创建的数据集。为了支持符号翻译阶段，我们也建立了一个新的名为BookGlossCorpus-CG的数据集，它将来自BookCorpus的日常英语句子转化为使用语法规则的手语符号。 我们的系统通过在符号之间平滑插值来实现自然流畅的动作衔接。与以往的研究（如How2Sign和Phoenix-2014T）不同，后者主要关注识别或仅利用一种类型的数据，我们所提出的流水线整合了音频、文本及动作，在单一框架中实现了从口语英语到逼真3D手语动画的全过程转换。
URL
https://arxiv.org/abs/2507.06530
