Abstract
Image segmentation is a critical task in microscopy, essential for accurately analyzing and interpreting complex visual data. This task can be performed using custom models trained on domain-specific datasets, transfer learning from pre-trained models, or foundational models that offer broad applicability. However, foundational models often present a considerable number of non-transparent tuning parameters that require extensive manual optimization, limiting their usability for real-time streaming data analysis. Here, we introduce a reward function-based optimization to fine-tune foundational models and illustrate this approach for SAM (Segment Anything Model) framework by Meta. The reward functions can be constructed to represent the physics of the imaged system, including particle size distributions, geometries, and other criteria. By integrating a reward-driven optimization framework, we enhance SAM's adaptability and performance, leading to an optimized variant, SAM$^{*}$, that better aligns with the requirements of diverse segmentation tasks and particularly allows for real-time streaming data segmentation. We demonstrate the effectiveness of this approach in microscopy imaging, where precise segmentation is crucial for analyzing cellular structures, material interfaces, and nanoscale features.
Abstract (translated)
图像分割是显微镜分析中的关键任务，对于准确解析和解释复杂的视觉数据至关重要。这项任务可以通过使用针对特定领域训练的自定义模型、从预训练模型迁移学习或使用具有广泛适用性的基础模型来完成。然而，基础模型通常包含大量不透明的调优参数，这些参数需要进行大量的手动优化，从而限制了它们在实时流数据分析中的实用性。 在这里，我们介绍了一种基于奖励函数的优化方法，用于微调基础模型，并通过Meta的SAM（Segment Anything Model）框架来说明这种方法。奖励函数可以构造为表示成像系统的物理特性，包括粒子尺寸分布、几何形状和其他标准。通过集成驱动奖励的优化框架，我们可以增强SAM的适应性和性能，从而产生一种优化变体SAM$^{*}$，使其更符合各种分割任务的需求，并特别允许实时流数据分割。 我们在显微镜成像中展示了这种方法的有效性，在这种应用中，精确的分割对于分析细胞结构、材料界面和纳米级特征至关重要。
URL
https://arxiv.org/abs/2509.07047
