Abstract
Accurate segmentation of neuronal cells in fluorescence microscopy is a fundamental task for quantitative analysis in computational neuroscience. However, it is significantly impeded by challenges such as the coexistence of densely packed and sparsely distributed cells, complex overlapping morphologies, and severe class imbalance. Conventional deep learning models often fail to preserve fine topological details or accurately delineate boundaries under these conditions. To address these limitations, we propose a novel deep learning framework, IMSAHLO (Integrating Multi-Scale Attention and Hybrid Loss Optimization), for robust and adaptive neuronal segmentation. The core of our model features Multi-Scale Dense Blocks (MSDBs) to capture features at various receptive fields, effectively handling variations in cell density, and a Hierarchical Attention (HA) mechanism that adaptively focuses on salient morphological features to preserve Region of Interest (ROI) boundary details. Furthermore, we introduce a novel hybrid loss function synergistically combining Tversky and Focal loss to combat class imbalance, alongside a topology-aware Centerline Dice (clDice) loss and a Contour-Weighted Boundary loss to ensure topological continuity and precise separation of adjacent cells. Large-scale experiments on the public Fluorescent Neuronal Cells (FNC) dataset demonstrate that our framework outperforms state-of-the-art architectures, achieving precision of 81.4%, macro F1 score of 82.7%, micro F1 score of 83.3%, and balanced accuracy of 99.5% on difficult dense and sparse cases. Ablation studies validate the synergistic benefits of multi-scale attention and hybrid loss terms. This work establishes a foundation for generalizable segmentation models applicable to a wide range of biomedical imaging modalities, pushing AI-assisted analysis toward high-throughput neurobiological pipelines.
Abstract (translated)
在荧光显微镜下对神经元细胞进行准确分割是计算神经科学中定量分析的一项基本任务。然而，由于密集排列和稀疏分布的细胞共存、复杂的重叠形态以及严重的类别不平衡等问题，这项工作受到了极大的阻碍。传统的深度学习模型在这种情况下往往难以保留精细的拓扑细节或精确地勾画边界。为了解决这些问题，我们提出了一种新的深度学习框架IMSAHLO（多尺度注意力与混合损失优化集成），用于稳健和适应性的神经元分割。我们的模型核心采用了多尺度密集块（MSDBs）来捕捉不同感受野的特征，有效处理细胞密度变化的问题，并且引入了层次化注意机制（HA）以自适应地聚焦于显著形态特征上，从而保留感兴趣区域（ROI）边界的细节。 此外，我们还提出了一种新的混合损失函数，该函数将Tversky和Focal loss相结合来对抗类别不平衡问题，同时结合拓扑感知的中心线Dice（clDice）损失和轮廓加权边界损失，以确保拓扑连续性和相邻细胞的精确分离。在公开的荧光神经元细胞（FNC）数据集上进行的大规模实验表明，我们的框架优于当前最先进的架构，在困难密集和稀疏案例中分别达到了81.4%的精度、82.7%的宏平均F1分数、83.3%的微平均F1分数以及99.5%的平衡准确性。消融研究验证了多尺度注意力与混合损失项之间的协同效应。 这项工作为适用于各种生物医学成像模式的一般化分割模型奠定了基础，推动了AI辅助分析向高通量神经生物学流水线方向发展。
URL
https://arxiv.org/abs/2601.11645
