Abstract
Sign Language (SL) enables two-way communication for the deaf and hard-of-hearing community, yet many sign languages remain under-resourced in the AI space. Sign Language Instruction Generation (SLIG) produces step-by-step textual instructions that enable non-SL users to imitate and learn SL gestures, promoting two-way interaction. We introduce BdSLIG, the first Bengali SLIG dataset, used to evaluate Vision Language Models (VLMs) (i) on under-resourced SLIG tasks, and (ii) on long-tail visual concepts, as Bengali SL is unlikely to appear in the VLM pre-training data. To enhance zero-shot performance, we introduce Sign Parameter-Infused (SPI) prompting, which integrates standard SL parameters, like hand shape, motion, and orientation, directly into the textual prompts. Subsuming standard sign parameters into the prompt makes the instructions more structured and reproducible than free-form natural text from vanilla prompting. We envision that our work would promote inclusivity and advancement in SL learning systems for the under-resourced communities.
Abstract (translated)
手语（SL）为聋人和听力障碍社区提供了双向交流的方式，然而许多手语在人工智能领域仍处于资源匮乏状态。手语指令生成（SLIG）能够产生分步骤的文本指令，使非手语使用者能够模仿并学习手语手势，从而促进双向互动。我们引入了BdSLIG，这是第一个孟加拉语手语指令生成数据集，用于评估视觉语言模型（VLMs）在资源匮乏的手语指令生成任务上的表现，并且针对长尾视觉概念进行测试，因为孟加拉语手语不太可能出现在VLM的预训练数据中。为了提升零样本性能，我们引入了融合手势参数提示（SPI），该方法将标准的手势参数，如手形、动作和朝向等直接融入文本提示中。将这些标准的手势参数嵌入到提示中，使得指令比普通的自由形式自然语言更具结构化且易于复现。我们希望我们的工作能够促进在资源匮乏社区中的手语学习系统的发展，并提高其包容性和先进性。
URL
https://arxiv.org/abs/2508.16076
