Abstract
Generating medical images from human-drawn free-hand sketches holds promise for various important medical imaging applications. Due to the extreme difficulty in collecting free-hand sketch data in the medical domain, most deep learning-based methods have been proposed to generate medical images from the synthesized sketches (e.g., edge maps or contours of segmentation masks from real images). However, these models often fail to generalize on the free-hand sketches, leading to unsatisfactory results. In this paper, we propose a practical free-hand sketch-to-image generation model called Sketch2MedI that learns to represent sketches in StyleGAN's latent space and generate medical images from it. Thanks to the ability to encode sketches into this meaningful representation space, Sketch2MedI only requires synthesized sketches for training, enabling a cost-effective learning process. Our Sketch2MedI demonstrates a robust generalization to free-hand sketches, resulting in high-quality and realistic medical image generations. Comparative evaluations of Sketch2MedI against the pix2pix, CycleGAN, UNIT, and U-GAT-IT models show superior performance in generating pharyngeal images, both quantitative and qualitative across various metrics.
Abstract (translated)
基于手绘的自由手绘图生成医学图像具有各种重要医学成像应用的潜力。由于在医疗领域中收集自由手绘数据非常困难,因此提出了许多基于深度学习的生成医学图像的方法,这些方法通常从合成图(例如,真实图像中的边缘图或分割掩码的轮廓)生成医学图像。然而,这些模型通常无法泛化到自由手绘图,导致不满意的结果。在本文中,我们提出了一个实用的自由手绘图到图像生成模型,称为Sketch2MedI,它学会了在StyleGAN的潜在空间中表示手绘图,并从中生成医学图像。由于能够将手绘图编码到这个有意义的表示空间中,Sketch2MedI仅需要合成图进行训练,从而实现了一种经济有效的学习过程。我们的Sketch2MedI在自由手绘图中表现出优异的泛化能力,从而能够生成高质量和真实的医学图像。Sketch2MedI与pix2pix、CycleGAN、UNIT和U-GAT-IT模型的比较评估显示,在生成咽喉图像方面,Sketch2MedI在量和质上均优于这些模型。
URL
https://arxiv.org/abs/2402.00353
