Abstract
Early diagnosis of renal cancer can greatly improve the survival rate of patients. Contrast-enhanced ultrasound (CEUS) is a cost-effective and non-invasive imaging technique and has become more and more frequently used for renal tumor diagnosis. However, the classification of benign and malignant renal tumors can still be very challenging due to the highly heterogeneous appearance of cancer and imaging artifacts. Our aim is to detect and classify renal tumors by integrating B-mode and CEUS-mode ultrasound videos. To this end, we propose a novel multi-modal ultrasound video fusion network that can effectively perform multi-modal feature fusion and video classification for renal tumor diagnosis. The attention-based multi-modal fusion module uses cross-attention and self-attention to extract modality-invariant features and modality-specific features in parallel. In addition, we design an object-level temporal aggregation (OTA) module that can automatically filter low-quality features and efficiently integrate temporal information from multiple frames to improve the accuracy of tumor diagnosis. Experimental results on a multicenter dataset show that the proposed framework outperforms the single-modal models and the competing methods. Furthermore, our OTA module achieves higher classification accuracy than the frame-level predictions. Our code is available at \url{this https URL}.
Abstract (translated)
早期诊断肾脏癌可以极大地提高患者的生存率。Contrast-enhanced ultrasound(CEUS)是一种成本效益高且非侵入性的成像技术,已经成为肾脏癌诊断越来越常用的方法。然而,良性和恶性肾脏癌的鉴别诊断由于癌症和成像误差的高度异质性仍然非常困难。我们的目标是通过整合B模式和CEUS模式超声波视频,有效地进行多模态特征融合和视频分类,以诊断肾脏癌。为此,我们提出了一种新的多模态超声波视频融合网络,可以进行多模态特征融合和视频分类,以肾脏癌诊断。基于注意力的多模态融合模块使用交叉注意力和自我注意力并行提取模态不相关特征和模态特定特征。此外,我们设计了一个对象级别的时间聚合(OTA)模块,可以自动过滤低质量特征并高效整合多个帧的时间信息,以提高癌症诊断的准确性。在一个多中心数据集的实验结果显示,该提出的框架比帧级预测实现了更高的分类准确性。我们的代码可访问\url{this https URL}。
URL
https://arxiv.org/abs/2307.07807
