Abstract
Deep learning underlies most modern approaches and tools in computer vision, including biomedical imaging. However, for interactive semantic segmentation (often called pixel classification in this context) and interactive object-level classification (object classification), feature-based shallow learning remains widely used. This is due to the diversity of data in this domain, the lack of large pretraining datasets, and the need for computational and label efficiency. In contrast, state-of-the-art tools for many other vision tasks in microscopy - most notably cellular instance segmentation - already rely on deep learning and have recently benefited substantially from vision foundation models (VFMs), particularly SAM. Here, we investigate whether VFMs can also improve pixel and object classification compared to current approaches. To this end, we evaluate several VFMs, including general-purpose models (SAM, SAM2, DINOv3) and domain-specific ones ($\mu$SAM, PathoSAM), in combination with shallow learning and attentive probing on five diverse and challenging datasets. Our results demonstrate consistent improvements over hand-crafted features and provide a clear pathway toward practical improvements. Furthermore, our study establishes a benchmark for VFMs in microscopy and informs future developments in this area.
Abstract (translated)
深度学习构成了计算机视觉（包括生物医学成像）中大多数现代方法与工具的基础。然而，在交互式语义分割（在此语境下常称为像素分类）和交互式对象级分类（对象分类）领域，基于特征的浅层学习仍被广泛使用。这源于该领域数据的多样性、缺乏大规模预训练数据集，以及对计算效率和标注效率的需求。相比之下，显微镜下许多其他视觉任务——尤其是细胞实例分割——的先进工具已依赖深度学习，并近期显著受益于视觉基础模型（VFMs），特别是SAM。在此研究中，我们探究VFMs是否也能在像素与对象分类方面相较于当前方法取得改进。为此，我们在五个多样且具有挑战性的数据集上评估了多个VFMs（包括通用模型SAM、SAM2、DINOv3及领域专用模型$\mu$SAM、PathoSAM），并结合浅层学习与注意力探测方法。结果表明，相较于手工设计特征，我们的方法实现了持续改进，并为实际应用提升提供了清晰路径。此外，本研究为显微镜领域的VFMs建立了基准，并为该领域未来发展提供了参考。
URL
https://arxiv.org/abs/2603.19802
