Abstract
Video surgery timelines are an important part of tool-assisted surgeries, as they allow surgeons to quickly focus on key parts of the procedure. Current methods involve the surgeon filling out a post-operation (OP) report, which is often vague, or manually annotating the surgical videos, which is highly time-consuming. Our proposed method sits between these two extremes: we aim to automatically create a surgical timeline and narrative directly from the surgical video. To achieve this, we employ a CLIP-based multi-modal framework that aligns surgical video frames with textual gesture descriptions. Specifically, we use the CLIP visual encoder to extract representations from surgical video frames and the text encoder to embed the corresponding gesture sentences into a shared embedding space. We then fine-tune the model to improve the alignment between video gestures and textual tokens. Once trained, the model predicts gestures and phases for video frames, enabling the construction of a structured surgical timeline. This approach leverages pretrained multi-modal representations to bridge visual gestures and textual narratives, reducing the need for manual video review and annotation by surgeons.
Abstract (translated)
视频手术时间线是辅助工具手术中非常重要的组成部分，因为它可以帮助外科医生快速聚焦于程序的关键部分。目前的方法包括外科医生填写术后报告，但这种报告通常不够详细；或者手动标注手术视频，这是一项耗时且繁琐的工作。我们提出了一种介于这两种方法之间的新方案：直接从手术视频自动创建手术时间线和叙述。 为了实现这一目标，我们采用了一个基于CLIP的多模态框架，该框架将手术视频帧与文本描述手势进行对齐。具体来说，我们使用CLIP视觉编码器来提取手术视频帧的表示，并利用文本编码器将其相应的手势句子嵌入到共享的嵌入空间中。然后，我们微调模型以改善视频手势和文本标记之间的对齐关系。一旦训练完成，该模型可以预测视频帧的手势及阶段，从而构建结构化的手术时间线。 这种方法通过利用预训练的多模态表示来连接视觉手势与文本叙述，极大地减少了外科医生对手术视频手动审查和标注的需求。
URL
https://arxiv.org/abs/2603.05732
