Abstract
Communication barriers pose significant challenges for individuals with hearing and speech impairments, often limiting their ability to effectively interact in everyday environments. This project introduces a real-time assistive technology solution that leverages advanced deep learning techniques to translate sign language gestures into textual and audible speech. By employing convolution neural networks (CNN) trained on the Sign Language MNIST dataset, the system accurately classifies hand gestures captured live via webcam. Detected gestures are instantaneously translated into their corresponding meanings and transcribed into spoken language using text-to-speech synthesis, thus facilitating seamless communication. Comprehensive experiments demonstrate high model accuracy and robust real-time performance with some latency, highlighting the system's practical applicability as an accessible, reliable, and user-friendly tool for enhancing the autonomy and integration of sign language users in diverse social settings.
Abstract (translated)
沟通障碍对听障和言语障碍人士构成了重大挑战，往往限制了他们在日常环境中有效互动的能力。本项目介绍了一种实时辅助技术解决方案，该方案利用先进的深度学习技术将手语手势翻译成文本和语音输出。通过使用在Sign Language MNIST数据集上训练的卷积神经网络（CNN），系统能够准确地识别并通过摄像头捕获的手势进行分类。检测到的手势会即时转换为其对应的含义，并通过文字转语音合成技术转化为口语，从而实现无缝交流。 全面的实验表明，该模型具有高精度和强大的实时性能，尽管存在一些延迟问题，但仍突显了作为无障碍、可靠且用户友好的工具在各种社交环境中提升手语使用者自主性和融入度的应用潜力。
URL
https://arxiv.org/abs/2508.12713
