Abstract
Real-time recognition and prediction of surgical activities are fundamental to advancing safety and autonomy in robot-assisted surgery. This paper presents a multimodal transformer architecture for real-time recognition and prediction of surgical gestures and trajectories based on short segments of kinematic and video data. We conduct an ablation study to evaluate the impact of fusing different input modalities and their representations on gesture recognition and prediction performance. We perform an end-to-end assessment of the proposed architecture using the JHU-ISI Gesture and Skill Assessment Working Set (JIGSAWS) dataset. Our model outperforms the state-of-the-art (SOTA) with 89.5\% accuracy for gesture prediction through effective fusion of kinematic features with spatial and contextual video features. It achieves the real-time performance of 1.1-1.3ms for processing a 1-second input window by relying on a computationally efficient model.
Abstract (translated)
实时识别和预测手术活动是推动机器人辅助手术安全性和自主性的基础。本文提出了一种基于短动作和视频数据的小段运动和视频数据的 multimodal Transformer 架构,用于实时识别和预测手术手势和轨迹。我们进行了一项消融研究,以评估将不同输入模块及其表示集成到手势识别和预测性能中的影响。我们使用 JHU-ISI Gesture and Skill Assessment Working Set (JIGSAWS) 数据集对所提出的架构进行了端到端评估。我们的模型在通过有效地融合运动特征和空间上下文视频特征来提高手势预测准确率的基础上,实现了与最先进水平(SOTA)的 89.5% 的准确率。它能在依赖计算效率模型的情况下,实现对 1 秒输入窗口的实时处理,并达到 1.1-1.3ms 的实时性能。
URL
https://arxiv.org/abs/2403.06705
