Abstract
Multimodal medical imaging plays a pivotal role in clinical diagnosis and research, as it combines information from various imaging modalities to provide a more comprehensive understanding of the underlying pathology. Recently, deep learning-based multimodal fusion techniques have emerged as powerful tools for improving medical image classification. This review offers a thorough analysis of the developments in deep learning-based multimodal fusion for medical classification tasks. We explore the complementary relationships among prevalent clinical modalities and outline three main fusion schemes for multimodal classification networks: input fusion, intermediate fusion (encompassing single-level fusion, hierarchical fusion, and attention-based fusion), and output fusion. By evaluating the performance of these fusion techniques, we provide insight into the suitability of different network architectures for various multimodal fusion scenarios and application domains. Furthermore, we delve into challenges related to network architecture selection, handling incomplete multimodal data management, and the potential limitations of multimodal fusion. Finally, we spotlight the promising future of Transformer-based multimodal fusion techniques and give recommendations for future research in this rapidly evolving field.
Abstract (translated)
多模态医疗影像在临床诊断和研究中扮演着至关重要的角色，因为它结合了各种影像模态的信息，提供更全面的病理解剖学理解。近年来，基于深度学习的多模态融合技术已成为提高医学图像分类的强大工具。本文对基于深度学习的多模态融合在医学分类任务的发展进行了全面的分析。我们探讨了主要临床模态之间的互补关系，并提出了三种主要的融合方案：输入融合、中间融合（包括单层融合、层次融合和基于注意力的融合）和输出融合。通过评估这些融合技术的性能，我们提供了对各种多模态融合场景和应用领域的适用网络架构的洞察。此外，我们还深入探讨了与网络架构选择、处理不完整的多模态数据管理以及多模态融合的潜在限制相关的问题。最后，我们重点关注了基于Transformer的多模态融合技术的光明未来，并给未来在这个快速发展的领域的研究提出了建议。
URL
https://arxiv.org/abs/2404.15022
