Abstract
Accurate segmentation of regions of interest in biomedical images holds substantial value in image analysis. Although several foundation models for biomedical segmentation have currently achieved excellent performance on certain datasets, they typically demonstrate sub-optimal performance on unseen domain data. We owe the deficiency to lack of vision-language knowledge before segmentation. Multimodal Large Language Models (MLLMs) bring outstanding understanding and reasoning capabilities to multimodal tasks, which inspires us to leverage MLLMs to inject Vision-Language Knowledge (VLK), thereby enabling vision models to demonstrate superior generalization capabilities on cross-domain datasets. In this paper, we propose using MLLMs to guide SAM in learning microscopy crose-domain data, unifying Segment Anything in Microscopy, named uLLSAM. Specifically, we propose the Vision-Language Semantic Alignment (VLSA) module, which injects VLK into Segment Anything Model (SAM). We find that after SAM receives global VLK prompts, its performance improves significantly, but there are deficiencies in boundary contour perception. Therefore, we further propose Semantic Boundary Regularization (SBR) to prompt SAM. Our method achieves performance improvements of 7.71% in Dice and 12.10% in SA across 9 in-domain microscopy datasets, achieving state-of-the-art performance. Our method also demonstrates improvements of 6.79% in Dice and 10.08% in SA across 10 out-ofdomain datasets, exhibiting strong generalization capabilities. Code is available at this https URL.
Abstract (translated)
在生物医学图像中精确分割感兴趣区域对图像分析具有重要价值。尽管目前有一些用于生物医学分割的基础模型已经在某些数据集上取得了优秀的性能，但它们通常在未见过的数据域（unseen domain data）上的表现较差。我们认为这一不足主要是由于缺乏视觉-语言知识（Vision-Language Knowledge, VLK）。多模态大型语言模型（Multimodal Large Language Models, MLLMs）在多模态任务中带来了出色的理解和推理能力，这激发了我们利用MLLMs注入VLK的想法，从而使视觉模型能够在跨域数据集上表现出色的泛化能力。在这篇论文中，我们提出了一种方法，即使用MLLMs来指导SAM（Segment Anything Model）学习显微图像跨域数据，命名为uLLSAM（unifying Segment Anything in Microscopy）。具体来说，我们提出了视觉-语言语义对齐（Vision-Language Semantic Alignment, VLSA）模块，该模块将VLK注入到SAM中。我们发现，在SAM接收到全局VLK提示后，其性能显著提高，但存在边界轮廓感知方面的不足。因此，我们进一步提出了一种语义边界正则化（Semantic Boundary Regularization, SBR），用于引导SAM。我们的方法在9个内域显微图像数据集上实现了Dice指标提高了7.71%，SA（可能指的是Segment Anything模型的评价标准之一）指标提高了12.10%的成绩，达到了最先进的性能水平。此外，在10个外域数据集中，我们的方法分别使Dice和SA指标提升了6.79%和10.08%，展示了强大的泛化能力。代码可在提供的链接中获取。
URL
https://arxiv.org/abs/2505.10769
