Abstract
While foundation models have advanced surgical video analysis, current approaches rely predominantly on pixel-level reconstruction objectives that waste model capacity on low-level visual details - such as smoke, specular reflections, and fluid motion - rather than semantic structures essential for surgical understanding. We present UniSurg, a video-native foundation model that shifts the learning paradigm from pixel-level reconstruction to latent motion prediction. Built on the Video Joint Embedding Predictive Architecture (V-JEPA), UniSurg introduces three key technical innovations tailored to surgical videos: 1) motion-guided latent prediction to prioritize semantically meaningful regions, 2) spatiotemporal affinity self-distillation to enforce relational consistency, and 3) feature diversity regularization to prevent representation collapse in texture-sparse surgical scenes. To enable large-scale pretraining, we curate UniSurg-15M, the largest surgical video dataset to date, comprising 3,658 hours of video from 50 sources across 13 anatomical regions. Extensive experiments across 17 benchmarks demonstrate that UniSurg significantly outperforms state-of-the-art methods on surgical workflow recognition (+14.6% F1 on EgoSurgery, +10.3% on PitVis), action triplet recognition (39.54% mAP-IVT on CholecT50), skill assessment, polyp segmentation, and depth estimation. These results establish UniSurg as a new standard for universal, motion-oriented surgical video understanding.
Abstract (translated)
尽管基础模型在手术视频分析方面取得了进展，但目前的方法主要依赖于像素级别的重构目标，这导致模型容量被浪费在低级视觉细节（如烟雾、镜面反射和液体流动）上，而不是对理解手术至关重要的语义结构。我们介绍了UniSurg，这是一种基于视频的基础模型，它将学习范式从像素级别重建转向潜在的运动预测。UniSurg建立在视频联合嵌入预测架构(V-JEPA)的基础上，并为手术视频引入了三项关键技术创新： 1. **以运动引导潜在预测**：优先考虑语义上有意义的区域。 2. **时空亲和自我蒸馏**：强制执行关系一致性。 3. **特征多样性正则化**：防止在纹理稀疏的手术场景中出现表示崩溃。 为了实现大规模预训练，我们整理了UniSurg-15M，这是迄今为止最大的手术视频数据集，包含来自50个来源、跨越13个解剖区域的3,658小时视频。通过17项基准测试的大规模实验表明，UniSurg在手术流程识别（EgoSurgery上的F1分数提高14.6%，PitVis上提高10.3%）、动作三元组识别（CholecT50上的mAP-IVT分数为39.54%）、技能评估、息肉分割和深度估计方面，均显著超越了现有最佳方法。这些结果确立了UniSurg作为通用且以运动为导向的手术视频理解新标准的地位。
URL
https://arxiv.org/abs/2602.05638
