Abstract
Medical Visual Question Answering (MedVQA), which offers language responses to image-based medical inquiries, represents a challenging task and significant advancement in healthcare. It assists medical experts to swiftly interpret medical images, thereby enabling faster and more accurate diagnoses. However, the model interpretability and transparency of existing MedVQA solutions are often limited, posing challenges in understanding their decision-making processes. To address this issue, we devise a semi-automated annotation process to streamlining data preparation and build new benchmark MedVQA datasets R-RAD and R-SLAKE. The R-RAD and R-SLAKE datasets provide intermediate medical decision-making rationales generated by multimodal large language models and human annotations for question-answering pairs in existing MedVQA datasets, i.e., VQA-RAD and SLAKE. Moreover, we design a novel framework which finetunes lightweight pretrained generative models by incorporating medical decision-making rationales into the training process. The framework includes three distinct strategies to generate decision outcomes and corresponding rationales, thereby clearly showcasing the medical decision-making process during reasoning. Extensive experiments demonstrate that our method can achieve an accuracy of 83.5% on R-RAD and 86.3% on R-SLAKE, significantly outperforming existing state-of-the-art baselines. Dataset and code will be released.
Abstract (translated)
医学视觉问题回答（MedVQA）是一种为图像医学查询提供语言回答的挑战性任务，并在医疗保健领域取得了显著的进展。它帮助医疗专家迅速解释医学图像，从而实现更快更准确的诊断。然而，现有MedVQA解决方案的可解释性和透明度往往有限，这使得理解其决策过程具有挑战性。为解决这个问题，我们设计了一个半自动化的注释过程，以简化数据准备并构建新的MedVQA数据集R-RAD和R-SLAKE。R-RAD和R-SLAKE数据集提供了现有MedVQA数据集中的多模态大型语言模型和人类标注的交互式问题答案对的中间医疗决策过程。此外，我们还设计了一个新框架，通过将医学决策过程融入训练过程来优化轻量级预训练生成模型。该框架包括生成决策结果和相应理由的三种不同的策略，从而在推理过程中清晰地展示医学决策过程。大量实验证明，我们的方法可以在R-RAD和R-SLAKE上实现83.5%的准确度，显著优于现有状态下的最先进基线。数据集和代码将发布。
URL
https://arxiv.org/abs/2404.12372
