Abstract
Cytopathology report generation is a necessary step for the standardized examination of pathology images. However, manually writing detailed reports brings heavy workloads for pathologists. To improve efficiency, some existing works have studied automatic generation of cytopathology reports, mainly by applying image caption generation frameworks with visual encoders originally proposed for natural images. A common weakness of these works is that they do not explicitly model the structural information among cells, which is a key feature of pathology images and provides significant information for making diagnoses. In this paper, we propose a novel graph-based framework called GNNFormer, which seamlessly integrates graph neural network (GNN) and Transformer into the same framework, for cytopathology report generation. To the best of our knowledge, GNNFormer is the first report generation method that explicitly models the structural information among cells in pathology images. It also effectively fuses structural information among cells, fine-grained morphology features of cells and background features to generate high-quality reports. Experimental results on the NMI-WSI dataset show that GNNFormer can outperform other state-of-the-art baselines.
Abstract (translated)
细胞病理学报告生成是标准化病理学图像检查的必要步骤。然而,手动撰写详细报告给病理学家带来了巨大的工作负担。为了提高效率,一些现有工作已经研究了自动生成细胞病理学报告的方法,主要是通过应用最初为自然图像所设计的可视化编码框架来实现图像标题生成框架。这些工作的一个共同弱点是它们并不明确 Modeling 细胞之间的结构信息,这是病理学图像的一个关键特征,并为诊断提供重要信息。在本文中,我们提出了一种新的基于图的结构主义框架,称为 GNN former,它将图神经网络 (GNN) 和变分自编码器无缝集成到同一个框架中,用于细胞病理学报告生成。据我们所知,GNN former 是第一种明确 Modeling 病理学图像中细胞之间的结构信息的报告生成方法。它还有效地融合细胞之间的结构信息、细胞细粒度形态特征和背景特征,生成高质量的报告。在NMI-WSI数据集上的实验结果显示,GNN former 可以与其他先进的基准方法相比表现更好。
URL
https://arxiv.org/abs/2303.09956
