Abstract
Precise surgical interventions are vital to patient safety, and advanced enhancement algorithms have been developed to assist surgeons in decision-making. Despite significant progress, these algorithms are typically designed for single tasks in specific scenarios, limiting their effectiveness in complex real-world situations. To address this limitation, we propose SurgVisAgent, an end-to-end intelligent surgical vision agent built on multimodal large language models (MLLMs). SurgVisAgent dynamically identifies distortion categories and severity levels in endoscopic images, enabling it to perform a variety of enhancement tasks such as low-light enhancement, overexposure correction, motion blur elimination, and smoke removal. Specifically, to achieve superior surgical scenario understanding, we design a prior model that provides domain-specific knowledge. Additionally, through in-context few-shot learning and chain-of-thought (CoT) reasoning, SurgVisAgent delivers customized image enhancements tailored to a wide range of distortion types and severity levels, thereby addressing the diverse requirements of surgeons. Furthermore, we construct a comprehensive benchmark simulating real-world surgical distortions, on which extensive experiments demonstrate that SurgVisAgent surpasses traditional single-task models, highlighting its potential as a unified solution for surgical assistance.
Abstract (translated)
精确的外科手术干预对于患者安全至关重要，为此已经开发出先进的增强算法来帮助外科医生做出决策。尽管取得了显著进展，但这些算法通常仅针对特定场景下的单一任务而设计，因此在复杂的现实情况中其有效性受到限制。为了解决这一局限性，我们提出了SurgVisAgent，这是一个基于多模态大型语言模型（MLLM）构建的端到端智能外科手术视觉代理。SurgVisAgent能够动态识别内窥镜图像中的失真类型和严重程度，并执行多种增强任务，例如低光增强、过曝校正、运动模糊消除以及烟雾去除。 为了实现对复杂外科场景的卓越理解，我们设计了一个提供领域特定知识的先验模型。此外，通过上下文少量样本学习（in-context few-shot learning）和链式思维推理（CoT），SurgVisAgent能够根据各种失真类型和严重程度定制图像增强方案，从而满足外科医生多样化的实际需求。 为了验证这一方法的有效性，我们构建了一个全面的基准测试集，该集合模拟了真实世界中的外科手术失真。实验结果表明，与传统的单一任务模型相比，SurgVisAgent表现更佳，突显了其作为统一外科辅助解决方案的巨大潜力。
URL
https://arxiv.org/abs/2507.02252
