Abstract
Robot-assisted surgery (RAS) has become a critical paradigm in modern surgery, promoting patient recovery and reducing the burden on surgeons through minimally invasive approaches. To fully realize its potential, however, a precise understanding of the visual data generated during surgical procedures is essential. Previous studies have predominantly focused on single-task approaches, but real surgical scenes involve complex temporal dynamics and diverse instrument interactions that limit comprehensive understanding. Moreover, the effective application of multi-task learning (MTL) requires sufficient pixel-level segmentation data, which are difficult to obtain due to the high cost and expertise required for annotation. In particular, long-term annotations such as phases and steps are available for every frame, whereas short-term annotations such as surgical instrument segmentation and action detection are provided only for key frames, resulting in a significant temporal-spatial imbalance. To address these challenges, we propose a novel framework that combines optical flow-based segmentation label interpolation with multi-task learning. optical flow estimated from annotated key frames is used to propagate labels to adjacent unlabeled frames, thereby enriching sparse spatial supervision and balancing temporal and spatial information for training. This integration improves both the accuracy and efficiency of surgical scene understanding and, in turn, enhances the utility of RAS.
Abstract (translated)
机器人辅助手术（RAS）已成为现代外科手术中的关键范式，通过微创方法促进患者康复并减轻外科医生的负担。然而，为了充分发挥其潜力，准确理解手术过程中生成的视觉数据至关重要。以往的研究主要集中在单一任务的方法上，但实际情况中手术场景涉及复杂的时序动态和多样化的器械交互，这限制了对其全面的理解。此外，多任务学习（MTL）的有效应用需要充足的像素级分割数据，而由于标注所需的高昂成本和专业知识，这些数据难以获取。特别是，长时间的注释如阶段和步骤为每一帧提供，而短期注释如手术器械分割和动作检测仅针对关键帧给出，导致显著的时间-空间不平衡。 为了应对这些挑战，我们提出了一种新的框架，该框架结合了基于光学流的分段标签插值与多任务学习。通过从标注的关键帧中估计出的光学流，可以将标签传播到相邻未标记的帧上，从而丰富稀疏的空间监督并平衡训练中的时间和空间信息。这种集成不仅提高了手术场景理解的准确性和效率，而且增强了RAS的应用价值。
URL
https://arxiv.org/abs/2509.18802
