Abstract
Understanding the fine-grained articulation of human hands is critical in high-stakes settings such as robot-assisted surgery, chip manufacturing, and AR/VR-based human-AI interaction. Despite achieving near-human performance on general vision-language benchmarks, current vision-language models (VLMs) struggle with fine-grained spatial reasoning, especially in interpreting complex and articulated hand poses. We introduce HandVQA, a large-scale diagnostic benchmark designed to evaluate VLMs' understanding of detailed hand anatomy through visual question answering. Built upon high-quality 3D hand datasets (FreiHAND, InterHand2.6M, FPHA), our benchmark includes over 1.6M controlled multiple-choice questions that probe spatial relationships between hand joints, such as angles, distances, and relative positions. We evaluate several state-of-the-art VLMs (LLaVA, DeepSeek and Qwen-VL) in both base and fine-tuned settings, using lightweight fine-tuning via LoRA. Our findings reveal systematic limitations in current models, including hallucinated finger parts, incorrect geometric interpretations, and poor generalization. HandVQA not only exposes these critical reasoning gaps but provides a validated path to improvement. We demonstrate that the 3D-grounded spatial knowledge learned from our benchmark transfers in a zero-shot setting, significantly improving accuracy of model on novel downstream tasks like hand gesture recognition (+10.33%) and hand-object interaction (+2.63%).
Abstract (translated)
深入理解人手的精细关节运动，在机器人辅助手术、芯片制造以及基于AR/VR的人机交互等高风险场景中至关重要。尽管当前视觉语言模型（VLMs）在通用视觉语言基准测试中已接近人类表现，但它们仍难以胜任精细的空间推理任务，尤其在解读复杂且多关节的手部姿态时。我们推出了HandVQA——一个大规模诊断性基准，旨在通过视觉问答评估VLMs对手部精细解剖结构的理解。该基准基于高质量3D手部数据集（FreiHAND、InterHand2.6M、FPHA），包含超过160万道受控的多项选择题，用于探测手部关节间的空间关系，如角度、距离和相对位置。我们使用轻量级LoRA微调方法，在基础模型和微调后两种设置下，对多个前沿视觉语言模型（LLaVA、DeepSeek、Qwen-VL）进行了评估。研究发现当前模型存在系统性局限：包括虚构手指部件、几何解释错误以及泛化能力不足。HandVQA不仅揭示了这些关键推理缺陷，还提供了经验证的改进路径。我们证明，从该基准学到的3D空间知识能够实现零样本迁移，显著提升模型在新任务上的准确率：手势识别提升10.33%，手物交互识别提升2.63%。
URL
https://arxiv.org/abs/2603.26362
