Abstract
Surgical video understanding is a crucial prerequisite for advancing Computer-Assisted Surgery. While vision-language models (VLMs) have recently been applied to the surgical domain, existing surgical vision-language datasets lack in capturing and evaluating complex, interleaved spatial-temporal dynamics. Creating large scale datasets that accurately represent fine-grained spatial-temporal relationships in surgical videos is challenging due to costly manual annotations or error-prone generation using large language models. To address this gap, we introduce the SurgSTU-Pipeline, a deterministic generation pipeline featuring temporal and spatial continuity filtering to reliably create surgical datasets for fine-grained spatial-temporal multimodal understanding. Applying this pipeline to publicly available surgical datasets, we create the SurgSTU dataset, comprising 7515 video clips densely extended with 150k fine-grained spatial-temporal question-answer samples. Our comprehensive evaluation shows that while state-of-the-art generalist VLMs struggle in zero-shot settings, their spatial-temporal capabilities can be improved through in-context learning. A fine-tuned VLM on the SurgSTU training dataset achieves highest performance among all spatial-temporal tasks, validating the dataset's efficacy to improve spatial-temporal understanding of VLMs in surgical videos. Code will be made publicly available.
Abstract (translated)
手术视频理解是推进计算机辅助手术的关键前提。尽管视觉语言模型（VLMs）近期已被应用于外科领域，但现有手术视觉语言数据集在捕捉和评估复杂交织的时空动态方面存在不足。由于人工标注成本高昂，或使用大语言模型生成时易产生错误，创建能准确表征手术视频中细粒度时空关系的大规模数据集极具挑战。为填补此空白，我们提出了SurgSTU-Pipeline——一种具备时空连续性过滤功能的确定性生成流程，可可靠地创建用于细粒度时空多模态理解的手术数据集。通过将该流程应用于公开手术数据集，我们构建了SurgSTU数据集，该数据集包含7515个视频片段，并密集扩展了15万条细粒度时空问答样本。全面评估表明，尽管最先进的通用视觉语言模型在零样本设置下表现欠佳，但其时空能力可通过上下文学习得到提升。在SurgSTU训练数据集上微调的视觉语言模型在所有时空任务中达到最高性能，验证了该数据集对提升视觉语言模型手术视频时空理解能力的有效性。代码将公开提供。
URL
https://arxiv.org/abs/2604.00784
