Paper Reading AI Learner

Style Transfer Enabled Sim2Real Framework for Efficient Learning of Robotic Ultrasound Image Analysis Using Simulated Data

2023-05-16 05:01:16
Keyu Li, Xinyu Mao, Chengwei Ye, Ang Li, Yangxin Xu, Max Q.-H. Meng

Abstract

Robotic ultrasound (US) systems have shown great potential to make US examinations easier and more accurate. Recently, various machine learning techniques have been proposed to realize automatic US image interpretation for robotic US acquisition tasks. However, obtaining large amounts of real US imaging data for training is usually expensive or even unfeasible in some clinical applications. An alternative is to build a simulator to generate synthetic US data for training, but the differences between simulated and real US images may result in poor model performance. This work presents a Sim2Real framework to efficiently learn robotic US image analysis tasks based only on simulated data for real-world deployment. A style transfer module is proposed based on unsupervised contrastive learning and used as a preprocessing step to convert the real US images into the simulation style. Thereafter, a task-relevant model is designed to combine CNNs with vision transformers to generate the task-dependent prediction with improved generalization ability. We demonstrate the effectiveness of our method in an image regression task to predict the probe position based on US images in robotic transesophageal echocardiography (TEE). Our results show that using only simulated US data and a small amount of unlabelled real data for training, our method can achieve comparable performance to semi-supervised and fully supervised learning methods. Moreover, the effectiveness of our previously proposed CT-based US image simulation method is also indirectly confirmed.

Abstract (translated)

机器人超声波(US)系统已经展现出了让超声波检查更加容易和更准确的潜力。近年来,已经提出了多种机器学习技术来实现机器人超声波摄取任务中的自动超声波图像解释。然而,获取大量的真实超声波图像用于训练通常非常昂贵或在某些临床应用程序中甚至不可行。一种替代方法是建立一个模拟器来生成合成的超声波数据用于训练,但模拟和真实超声波图像之间的差异可能会导致模型性能不佳。本文提出了一个Sim2Real框架,以高效地学习机器人超声波图像分析任务,仅基于模拟数据进行实际部署。基于无监督比较学习的样式转移模块被提出,并用作预处理步骤,将真实的超声波图像转换为模拟风格。此后,一个与任务相关的模型被设计,结合卷积神经网络和视觉转换器,生成任务相关的增强泛化能力的预测。我们证明了我们的方法在图像回归任务中,以预测机器人超声波心动图(TEE)中探针位置的超声波图像,的方法的有效性。我们的结果表明,仅使用模拟的超声波数据和少量的未标记的真实数据用于训练,我们的方法可以与半监督和完全监督学习方法相媲美。此外,我们之前提出的基于CT的超声波图像模拟方法的效果也间接得到了确认。

URL

https://arxiv.org/abs/2305.09169

PDF

https://arxiv.org/pdf/2305.09169.pdf


Tags
3D Action Action_Localization Action_Recognition Activity Adversarial Agent Attention Autonomous Bert Boundary_Detection Caption Chat Classification CNN Compressive_Sensing Contour Contrastive_Learning Deep_Learning Denoising Detection Dialog Diffusion Drone Dynamic_Memory_Network Edge_Detection Embedding Embodied Emotion Enhancement Face Face_Detection Face_Recognition Facial_Landmark Few-Shot Gait_Recognition GAN Gaze_Estimation Gesture Gradient_Descent Handwriting Human_Parsing Image_Caption Image_Classification Image_Compression Image_Enhancement Image_Generation Image_Matting Image_Retrieval Inference Inpainting Intelligent_Chip Knowledge Knowledge_Graph Language_Model Matching Medical Memory_Networks Multi_Modal Multi_Task NAS NMT Object_Detection Object_Tracking OCR Ontology Optical_Character Optical_Flow Optimization Person_Re-identification Point_Cloud Portrait_Generation Pose Pose_Estimation Prediction QA Quantitative Quantitative_Finance Quantization Re-identification Recognition Recommendation Reconstruction Regularization Reinforcement_Learning Relation Relation_Extraction Represenation Represenation_Learning Restoration Review RNN Salient Scene_Classification Scene_Generation Scene_Parsing Scene_Text Segmentation Self-Supervised Semantic_Instance_Segmentation Semantic_Segmentation Semi_Global Semi_Supervised Sence_graph Sentiment Sentiment_Classification Sketch SLAM Sparse Speech Speech_Recognition Style_Transfer Summarization Super_Resolution Surveillance Survey Text_Classification Text_Generation Tracking Transfer_Learning Transformer Unsupervised Video_Caption Video_Classification Video_Indexing Video_Prediction Video_Retrieval Visual_Relation VQA Weakly_Supervised Zero-Shot