Abstract
Surgical video understanding is essential for computer-assisted interventions, yet existing surgical foundation models remain constrained by limited data scale, procedural diversity, and inconsistent evaluation, often lacking a reproducible training pipeline. We propose SurgRec, a scalable and reproducible pretraining recipe for surgical video understanding, instantiated with two variants: SurgRec-MAE and SurgRec-JEPA. We curate a large multi-source corpus of 10,535 videos and 214.5M frames spanning endoscopy, laparoscopy, cataract, and robotic surgery. Building on this corpus, we develop a unified pretraining pipeline with balanced sampling and standardize a reproducible benchmark across 16 downstream datasets and four clinical domains with consistent data splits. Across extensive comparisons against SSL baselines and vision-language models, SurgRec consistently achieves superior performance across downstream datasets. In contrast, VLMs prove unreliable for fine-grained temporal recognition, exhibiting both performance gaps and sensitivity to prompt phrasing. Our work provides a reproducible, scalable foundation for the community to build more general surgical video models. All code, models, and data will be publicly released.
Abstract (translated)
手术视频理解对于计算机辅助干预至关重要，然而现有手术基础模型仍受限于数据规模有限、流程多样性不足以及评估标准不统一等问题，往往缺乏可复现的训练流程。本文提出SurgRec，一种可扩展且可复现的手术视频理解预训练方案，具体实现两个变体：SurgRec-MAE与SurgRec-JEPA。我们构建了一个涵盖内窥镜、腹腔镜、白内障及机器人手术的大型多源语料库，包含10,535个视频与2.145亿帧图像。基于该语料库，我们开发了统一的预训练流程，采用平衡采样策略，并在涵盖16个下游数据集与四个临床领域的标准化可复现基准上统一数据划分。在与自监督学习基线及视觉语言模型的广泛对比中，SurgRec在所有下游数据集上均表现优异。相比之下，视觉语言模型在细粒度时序识别任务中可靠性不足，既存在性能差距，又对提示词表述敏感。本研究为社区构建更通用手术视频模型提供了可复现、可扩展的基础框架。所有代码、模型与数据将公开发布。
URL
https://arxiv.org/abs/2603.29966
