Abstract
Sign Language Production (SLP) is the process of converting the complex input text into a real video. Most previous works focused on the Text2Gloss, Gloss2Pose, Pose2Vid stages, and some concentrated on Prompt2Gloss and Text2Avatar stages. However, this field has made slow progress due to the inaccuracy of text conversion, pose generation, and the rendering of poses into real human videos in these stages, resulting in gradually accumulating errors. Therefore, in this paper, we streamline the traditional redundant structure, simplify and optimize the task objective, and design a new sign language generative model called Stable Signer. It redefines the SLP task as a hierarchical generation end-to-end task that only includes text understanding (Prompt2Gloss, Text2Gloss) and Pose2Vid, and executes text understanding through our proposed new Sign Language Understanding Linker called SLUL, and generates hand gestures through the named SLP-MoE hand gesture rendering expert block to end-to-end generate high-quality and multi-style sign language videos. SLUL is trained using the newly developed Semantic-Aware Gloss Masking Loss (SAGM Loss). Its performance has improved by 48.6% compared to the current SOTA generation methods.
Abstract (translated)
手语生成（SLP）是指将复杂的输入文本转换为真实视频的过程。以往的研究主要集中在Text2Gloss、Gloss2Pose和Pose2Vid等阶段，也有一些研究关注Prompt2Gloss和Text2Avatar阶段。然而，由于这些阶段中文字转换的不准确性、姿态生成以及姿态转化为真人视频时存在的误差逐渐累积，该领域的发展较为缓慢。 因此，在这篇论文中，我们简化了传统的冗余结构，优化并精简了任务目标，并设计了一个新的手语生成模型——Stable Signer。它将SLP任务重新定义为一个包含文本理解和姿势到视频转换的分层端到端任务（即Prompt2Gloss、Text2Gloss和Pose2Vid）。我们通过提出的Sign Language Understanding Linker (SLUL) 来执行文本理解，并利用名为SLP-MoE的手势渲染专家块来生成手势，从而实现高质量且多风格的手语视频的端到端生成。该模型中的SLUL使用新的感知词汇掩码损失（Semantic-Aware Gloss Masking Loss, SAGM Loss）进行训练，在当前最先进的生成方法中表现出48.6%的性能提升。
URL
https://arxiv.org/abs/2512.04048
