Abstract
Robotic-assisted surgery (RAS) is established in clinical practice, and automated surgical skill assessment utilizing multimodal data offers transformative potential for surgical analytics and education. However, developing effective multimodal methods remains challenging due to the task complexity, limited annotated datasets and insufficient techniques for cross-modal information fusion. Existing state-of-the-art relies exclusively on RGB video and only applies on dry-lab settings, failing to address the significant domain gap between controlled simulation and real clinical cases, where the surgical environment together with camera and tissue motion introduce substantial complexities. This work introduces SurgFusion-Net and Divergence Regulated Attention (DRA), an innovative fusion strategy for multimodal surgical skill assessment. We contribute two first-of-their-kind clinical datasets: the RAH-skill dataset containing 279,691 RGB frames from 37 videos of Robot-assisted Hysterectomy (RAH), and the RARP-skill dataset containing 70,661 RGB frames from 33 videos of Robot-Assisted Radical Prostatectomy (RARP). Both datasets include M-GEARS skill annotations, corresponding optical flow and tool segmentation masks. DRA incorporates adaptive dual attention and diversity-promoting multi-head attention to fuse multimodal information, from three modalities, based on surgical context, enhancing assessment accuracy and reliability. Validated on the JIGSAWS benchmark, RAH-skill, and RARP-skill datasets, our approach outperforms recent baselines with SCC improvements of 0.02 in LOSO, 0.04 in LOUO across JIGSAWS tasks, and 0.0538 and 0.0493 gains on RAH-skill and RARP-skill, respectively.
Abstract (translated)
机器人辅助手术（RAS）已广泛应用于临床实践中，利用多模态数据进行自动化外科技能评估在手术分析和教育方面具有变革性的潜力。然而，由于任务复杂性、标注数据集有限以及跨模态信息融合技术不足等原因，开发有效的多模态方法仍面临挑战。现有的最先进的技术仅依赖于RGB视频，并且只适用于模拟实验室环境，无法解决受控仿真与实际临床案例之间的显著领域差距问题，在实际手术环境中，摄像机和组织的运动带来了相当大的复杂性。 这项工作引入了SurgFusion-Net以及差异调节注意力（DRA），这是一种创新性的多模态融合策略用于外科技能评估。我们贡献了两个具有开创意义的临床数据集：包含279,691张RGB图像，来自37个机器人辅助子宫切除术(RAH)视频片段的RAH-skill数据集；以及包含70,661张RGB图像，来自33个机器人辅助前列腺根治性切除术(RARP)视频片段的RARP-skill数据集。这两个数据集均包括M-GEARS技能标注、对应的光学流和工具分割掩码。 DRA方法结合了自适应双注意力机制以及多头注意力以促进多样性，并根据手术上下文从三种模态中融合多模态信息，从而提高评估的准确性和可靠性。在JIGSAWS基准测试、RAH-skill数据集及RARP-skill数据集上验证后，我们的方法优于近期的基线，在LOSO（Leave-One-Subject-Out）和LOUO（Leave-One-User-Out）设置下分别取得了0.02和0.04 SCC改进，以及在JIGSAWS任务上的表现有所提高；同时对RAH-skill数据集和RARP-skill数据集的改进幅度分别为0.0538和0.0493。
URL
https://arxiv.org/abs/2603.00108
