Abstract
We investigate how both the adaptation of a generic foundation model via transfer learning and the integration of complementary modalities from the operating room (OR) can support surgical data science. To this end, we use V-JEPA as the single-modality foundation of a multimodal model for minimally invasive surgery support. We analyze how the model's downstream performance can benefit (a) from finetuning on unlabeled surgical video data and (b) from providing additional time-resolved data streams from the OR in a multimodal setup. In an in-house dataset of liver surgery videos, we analyze the tasks of predicting hospital length of stay and postoperative complications. In videos of the public HeiCo dataset, we analyze the task of surgical phase recognition. As a baseline, we apply pretrained V-JEPA to all tasks. We then finetune it on unlabeled, held-out videos to investigate its change in performance after domain adaptation. Following the idea of modular decision support networks, we integrate additional data streams from the OR by training a separate encoder to form a shared representation space with V-JEPA's embeddings. Our experiments show that finetuning on domain-specific data increases model performance. On the in-house data, integrating additional time-resolved data likewise benefits the model. On the HeiCo data, accuracy of the pretrained video-only, single-modality baseline setup is on par with the top-performing submissions of the EndoVis2017 challenge, while finetuning on domain-specific data increases accuracy further. Our results thus demonstrate how surgical data science can leverage public, generic foundation models. Likewise, they indicate the potential of domain adaptation and of integrating suitable complementary data streams from the OR. To support further research, we release our code and model weights at this https URL.
Abstract (translated)
我们研究了通过迁移学习调整通用基础模型以及整合来自手术室（OR）的互补模态如何支持外科数据科学。为此，我们将V-JEPA用作微创手术支持多模态模型的基础单模态模型。我们分析了该模型在以下两种情况下的下游性能如何受益：(a) 在未标注的手术视频数据上进行微调；以及(b) 通过提供来自OR的时间序列附加数据流，在多模态设置下运行。在一个内部肝脏手术视频数据集中，我们分析了预测住院时间长短和术后并发症的任务。在公共HeiCo数据集中的视频中，我们分析了识别外科阶段的任务。作为基线，我们将预训练的V-JEPA应用于所有任务。然后我们在未标注、保留的数据上对其进行微调，以研究其在领域适应后的性能变化。借鉴模块化决策支持网络的理念，通过训练一个单独的编码器来形成与V-JEPA嵌入共享的表示空间的方式，我们整合了来自OR的附加数据流。 我们的实验表明，在特定领域的数据上进行微调可以提高模型的表现。在内部数据集上，集成额外的时间序列数据同样有助于提升模型性能。而在HeiCo数据集上，预训练视频单一模态基线设置的准确性与EndoVis2017挑战赛中最佳提交结果相当，而针对特定领域数据进行微调可以进一步提高精度。 因此，我们的研究结果展示了外科数据科学如何利用公共、通用的基础模型。同时，它们表明了域适应和整合来自OR的合适互补数据流的潜在价值。为了支持后续的研究工作，我们在以下网址发布了我们的代码和模型权重：[此处应填写具体URL]。
URL
https://arxiv.org/abs/2509.06831
