Abstract
Numerous sign language datasets exist, yet they typically cover only a limited selection of the thousands of signs used globally. Moreover, creating diverse sign language datasets is an expensive and challenging task due to the costs associated with gathering a varied group of signers. Motivated by these challenges, we aimed to develop a solution that addresses these limitations. In this context, we focused on textually describing body movements from skeleton keypoint sequences, leading to the creation of a new dataset. We structured this dataset around AUTSL, a comprehensive isolated Turkish sign language dataset. We also developed a baseline model, SkelCap, which can generate textual descriptions of body movements. This model processes the skeleton keypoints data as a vector, applies a fully connected layer for embedding, and utilizes a transformer neural network for sequence-to-sequence modeling. We conducted extensive evaluations of our model, including signer-agnostic and sign-agnostic assessments. The model achieved promising results, with a ROUGE-L score of 0.98 and a BLEU-4 score of 0.94 in the signer-agnostic evaluation. The dataset we have prepared, namely the AUTSL-SkelCap, will be made publicly available soon.
Abstract (translated)
许多手语数据集存在，但它们通常只覆盖了全球数百万种手语中的一小部分。此外，创建多样手语数据集是一个昂贵且具有挑战性的任务，因为收集到一个多样群体手语者相关的费用。为了克服这些挑战，我们旨在开发一种解决方案，以解决这些限制。在这个背景下，我们关注从骨骼关键点序列中描述身体运动，从而创建了一个新的数据集。我们围绕AUTSL，一个全面的隔离土耳其手语数据集，构建了 this dataset。我们还开发了一个 baseline 模型，SkeletonCap，可以生成身体运动的文本描述。这个模型将骨骼关键点数据处理为向量，应用了全连接层进行嵌入，并利用了Transformer神经网络进行序列到序列建模。我们对我们的模型进行了广泛的评估，包括 signer-agnostic 和 sign-agnostic 评估。我们的模型取得了很好的效果，在 signer-agnostic 评估中的 ROUGE-L 分数为 0.98，BLEU-4 分数为 0.94。我们准备的数据集，即 AUTSL-SkeletonCap，很快将公开发布。
URL
https://arxiv.org/abs/2405.02977
