Abstract
This study presents a comprehensive review of the potential of multimodal deep learning (DL) in medical diagnosis, using COVID-19 as a case example. Motivated by the success of artificial intelligence applications during the COVID-19 pandemic, this research aims to uncover the capabilities of DL in disease screening, prediction, and classification, and to derive insights that enhance the resilience, sustainability, and inclusiveness of science, technology, and innovation systems. Adopting a systematic approach, we investigate the fundamental methodologies, data sources, preprocessing steps, and challenges encountered in various studies and implementations. We explore the architecture of deep learning models, emphasising their data-specific structures and underlying algorithms. Subsequently, we compare different deep learning strategies utilised in COVID-19 analysis, evaluating them based on methodology, data, performance, and prerequisites for future research. By examining diverse data types and diagnostic modalities, this research contributes to scientific understanding and knowledge of the multimodal application of DL and its effectiveness in diagnosis. We have implemented and analysed 11 deep learning models using COVID-19 image, text, and speech (ie, cough) data. Our analysis revealed that the MobileNet model achieved the highest accuracy of 99.97% for COVID-19 image data and 93.73% for speech data (i.e., cough). However, the BiGRU model demonstrated superior performance in COVID-19 text classification with an accuracy of 99.89%. The broader implications of this research suggest potential benefits for other domains and disciplines that could leverage deep learning techniques for image, text, and speech analysis.
Abstract (translated)
这项研究提供了多模态深度学习（DL）在医学诊断中潜在应用的全面回顾，以COVID-19为例。鉴于人工智能技术在新冠疫情期间的成功应用，本研究旨在揭示深度学习在疾病筛查、预测和分类方面的潜力，并从中获得有助于增强科学、技术和创新体系韧性、可持续性和包容性的见解。采用系统方法，我们探讨了各种研究与实施中遇到的基本方法论、数据来源、预处理步骤以及所面临的挑战。我们还探索了深度学习模型的架构，强调其特定于数据的结构及其基础算法。接下来，我们将比较在COVID-19分析中使用的不同深度学习策略，并根据方法学、数据、性能和未来研究的需求对其进行评估。 通过考察不同类型的数据及诊断模式，本研究为多模态应用下的DL科学理解和知识贡献了力量，并探讨其在诊断中的有效性。我们实施并分析了11种基于COVID-19图像、文本以及语音（即咳嗽）数据的深度学习模型。我们的分析表明，MobileNet模型对COVID-19图像数据实现了最高精度为99.97%，而针对语音数据（如咳嗽）的准确率达到了93.73%。然而，在COVID-19文本分类中，BiGRU模型表现出色，其准确性达到99.89%。 这项研究更广泛的含义在于，它可能对其他领域和学科产生潜在益处，这些领域和学科可以利用深度学习技术进行图像、文本以及语音分析。
URL
https://arxiv.org/abs/2501.09506
