Abstract
Recent advancements in Large Multimodal Models (LMMs) have attracted interest in their generalization capability with only a few samples in the prompt. This progress is particularly relevant to the medical domain, where the quality and sensitivity of data pose unique challenges for model training and application. However, the dependency on high-quality data for effective in-context learning raises questions about the feasibility of these models when encountering with the inevitable variations and errors inherent in real-world medical data. In this paper, we introduce MID-M, a novel framework that leverages the in-context learning capabilities of a general-domain Large Language Model (LLM) to process multimodal data via image descriptions. MID-M achieves a comparable or superior performance to task-specific fine-tuned LMMs and other general-domain ones, without the extensive domain-specific training or pre-training on multimodal data, with significantly fewer parameters. This highlights the potential of leveraging general-domain LLMs for domain-specific tasks and offers a sustainable and cost-effective alternative to traditional LMM developments. Moreover, the robustness of MID-M against data quality issues demonstrates its practical utility in real-world medical domain applications.
Abstract (translated)
近年来在大型多模态模型（LMMs）方面的进步引起了人们对它们在提示中仅几个样本时的泛化能力的关注。在医疗领域，数据质量和准确性对模型的训练和应用提出了独特的挑战。然而，对高质量数据的需求对于在真实世界医疗数据中实现有效的上下文学习提出了问题，这也使得这些模型在遇到真实世界医疗数据的固有变异性误差时具有可行性。在本文中，我们引入了MID-M，一种利用一般领域大型语言模型（LLM）的上下文学习能力处理多模态数据的框架。MID-M在任务特定细粒度调整的LLM和其他一般领域的模型上实现了与LLM相当或更好的性能，而无需进行广泛的领域特定训练或预训练。这突显了利用通用领域LLM进行领域特定任务的潜力，并为传统LMM发展提供了一种可持续且成本效益高的替代方案。此外，MID-M对数据质量问题的鲁棒性表明其在真实世界医疗领域应用的实用性。
URL
https://arxiv.org/abs/2405.01591
