Abstract
In surgical training for medical students, proficiency development relies on expert-led skill assessment, which is costly, time-limited, difficult to scale, and its expertise remains confined to institutions with available specialists. Automated AI-based assessment offers a viable alternative, but progress is constrained by the lack of datasets containing realistic trainee errors and the multi-view variability needed to train robust computer vision approaches. To address this gap, we present Surgical-Hands (SHands), a large-scale multi-view video dataset for surgical hand-gesture and error recognition for medical training. \textsc{SHands} captures linear incision and suturing using five RGB cameras from complementary viewpoints, performed by 52 participants (20 experts and 32 trainees), each completing three standardized trials per procedure. The videos are annotated at the frame level with 15 gesture primitives and include a validated taxonomy of 8 trainee error types, enabling both gesture recognition and error detection. We further define standardized evaluation protocols for single-view, multi-view, and cross-view generalization, and benchmark state-of-the-art deep learning models on the dataset. SHands is publicly released to support the development of robust and scalable AI systems for surgical training grounded in clinically curated domain knowledge.
Abstract (translated)
在医学生手术培训中，技能熟练度的发展依赖于专家主导的技能评估，但这种方式成本高昂、时间有限、难以规模化，且专业知识仅限于拥有专科医师的医疗机构。基于人工智能的自动化评估提供了一种可行的替代方案，然而其进展受限于缺乏包含真实学员错误的数据集，以及训练鲁棒计算机视觉方法所需的多视角变化。为填补这一空白，我们提出了Surgical-Hands（SHands）——一个用于手术手势与错误识别的大规模多视角视频数据集，旨在支持医学培训。SHands通过五个互补视角的RGB摄像头采集了线性切口与缝合操作，由52名参与者（20名专家与32名学员）完成，每人每项操作进行三次标准化试验。视频在帧级别标注了15种手势基元，并包含经过验证的8类学员错误分类体系，可同时支持手势识别与错误检测。我们还定义了单视角、多视角及跨视角泛化的标准化评估协议，并在该数据集上对前沿深度学习模型进行了基准测试。SHands已公开发布，旨在支持基于临床精选领域知识的、稳健且可扩展的手术培训AI系统发展。
URL
https://arxiv.org/abs/2603.26400
