Abstract
Gestures enable non-verbal human-robot communication, especially in noisy environments like agile production. Traditional deep learning-based gesture recognition relies on task-specific architectures using images, videos, or skeletal pose estimates as input. Meanwhile, Vision Foundation Models (VFMs) and Vision Language Models (VLMs) with their strong generalization abilities offer potential to reduce system complexity by replacing dedicated task-specific modules. This study investigates adapting such models for dynamic, full-body gesture recognition, comparing V-JEPA (a state-of-the-art VFM), Gemini Flash 2.0 (a multimodal VLM), and HD-GCN (a top-performing skeleton-based approach). We introduce NUGGET, a dataset tailored for human-robot communication in intralogistics environments, to evaluate the different gesture recognition approaches. In our experiments, HD-GCN achieves best performance, but V-JEPA comes close with a simple, task-specific classification head - thus paving a possible way towards reducing system complexity, by using it as a shared multi-task model. In contrast, Gemini struggles to differentiate gestures based solely on textual descriptions in the zero-shot setting, highlighting the need of further research on suitable input representations for gestures.
Abstract (translated)
手势使非言语的人机通信成为可能，特别是在像敏捷生产这样的嘈杂环境中。传统的基于深度学习的手势识别方法依赖于特定任务的架构，这些架构使用图像、视频或骨骼姿态估计作为输入。同时，视觉基础模型（VFMs）和视觉语言模型（VLMs），凭借其强大的泛化能力，提供了减少系统复杂性的潜力，通过替换专用的任务特定模块来实现这一点。这项研究调查了将此类模型适应于动态全身手势识别的方法，并比较了最先进的视觉基础模型V-JEPA、多模态视觉语言模型Gemini Flash 2.0和基于骨骼的顶级方法HD-GCN。我们引入了NUGGET数据集，该数据集专为物流环境中的人机通信而设计，用于评估不同的手势识别方法。在我们的实验中，HD-GCN取得了最佳性能，但V-JEPA通过使用简单的任务特定分类头也表现出色，这表明向更简单、多功能模型发展的可能性，以减少系统复杂性。相比之下，在零样本设置下仅根据文本描述区分手势时，Gemini面临困难，这强调了进一步研究适合手势的输入表示形式的重要性。
URL
https://arxiv.org/abs/2506.20795
