Abstract
This paper presents a real-time American Sign Language (ASL) recognition system utilizing a hybrid deep learning architecture combining 3D Convolutional Neural Networks (3D CNN) with Long Short-Term Memory (LSTM) networks. The system processes webcam video streams to recognize word-level ASL signs, addressing communication barriers for over 70 million deaf and hard-of-hearing individuals worldwide. Our architecture leverages 3D convolutions to capture spatial-temporal features from video frames, followed by LSTM layers that model sequential dependencies inherent in sign language gestures. Trained on the WLASL dataset (2,000 common words), ASL-LEX lexical database (~2,700 signs), and a curated set of 100 expert-annotated ASL signs, the system achieves F1-scores ranging from 0.71 to 0.99 across sign classes. The model is deployed on AWS infrastructure with edge deployment capability on OAK-D cameras for real-time inference. We discuss the architecture design, training methodology, evaluation metrics, and deployment considerations for practical accessibility applications.
Abstract (translated)
本文提出了一种利用混合深度学习架构（结合3D卷积神经网络(3D CNN)和长短期记忆(LSTM)网络）的实时美国手语(ASL)识别系统。该系统处理来自网络摄像头的视频流，以识别单词级别的ASL手势，旨在解决全球超过7000万聋人和听力障碍人士在沟通方面的障碍。 我们的架构利用3D卷积来捕捉视频帧中的空间-时间特征，并通过随后的LSTM层建模手语动作中固有的序列依赖性。该系统在WLASL数据集（2,000个常用单词）、ASL-LEX词汇数据库（约2,700个手势）和100个专家注释的手势集合上进行了训练，实现了从0.71到0.99的F1分数。 该模型部署在AWS基础设施上，并具备边缘计算能力，在OAK-D相机上进行实时推理。本文讨论了架构设计、培训方法、评估指标以及针对实际可访问性应用的部署考虑因素。
URL
https://arxiv.org/abs/2512.22177
