Abstract
Most existing robotic surgery systems adopt a human-in-the-loop paradigm, often with the surgeon directly teleoperating the robotic system. Adding intelligence to these robots would enable higher-level control, such as supervised autonomy or even full autonomy. However, artificial intelligence (AI) requires large amounts of training data, which is currently lacking. This work proposes SurgSync, a multi-modal data collection framework with offline and online synchronization to support training and real-time inference, respectively. The framework is implemented on a da Vinci Research Kit (dVRK) and introduces (1) dual-mode (online/offline-matching) synchronized recorders, (2) a modern stereo endoscope to achieve image quality on par with clinical systems, and (3) additional sensors such as a side-view camera and a novel capacitive contact sensor to provide ground truth contact data. The framework also incorporates a post-processing toolbox for tasks such as depth estimation, optical flow, and a practical kinematic reprojection method using Gaussian heatmap. User studies with participants of varying skill levels are performed with ex-vivo tissue to provide clinically realistic data, and a network for surgical skill assessment is employed to demonstrate utilization of the collected data. Through the user study experiments, we obtained a dataset of 214 validated instances across multiple canonical training tasks. All software and data are available at this http URL.
Abstract (translated)
大多数现有的机器人手术系统采用人在环路（human-in-the-loop）的模式，通常是由外科医生直接操控机器人系统。向这些机器人添加智能功能将能够实现更高层次的控制，比如监督自主性或完全自主性。然而，人工智能（AI）需要大量的训练数据，而目前这方面的数据严重不足。为此，本研究提出了一种名为SurgSync的多模态数据采集框架，该框架包括离线和在线同步两个部分，分别用于支持训练和实时推断。该框架在达芬奇机器人手术系统的研发套件（da Vinci Research Kit, dVRK）上实现，并引入了以下三个关键组件： 1. 双模式（在线/离线匹配）的同步记录器； 2. 一种现代立体内窥镜，能够在图像质量方面与临床系统相媲美； 3. 额外传感器，如侧视摄像头和创新的电容接触传感器，提供真实的接触数据。 此外，该框架还集成了一个后期处理工具包，用于执行深度估计、光学流等任务，并采用高斯热图（Gaussian heatmap）来实现实际运动学重投影方法。通过使用离体组织进行技能水平各异参与者的用户研究，获取了临床真实的数据；同时部署了一个手术技能评估网络以展示收集数据的利用情况。通过这些用户实验研究，我们获得了涵盖多个典型训练任务的214个验证实例的数据集。所有软件和数据均可在提供的网址上获得。
URL
https://arxiv.org/abs/2603.06919
