Abstract
Vision language models (VLMs) have recently emerged and gained the spotlight for their ability to comprehend the dual modality of image and textual data. VLMs such as LLaVA, ChatGPT-4, and Gemini have recently shown impressive performance on tasks such as natural image captioning, visual question answering (VQA), and spatial reasoning. Additionally, a universal segmentation model by Meta AI, Segment Anything Model (SAM) shows unprecedented performance at isolating objects from unforeseen images. Since medical experts, biologists, and materials scientists routinely examine microscopy or medical images in conjunction with textual information in the form of captions, literature, or reports, and draw conclusions of great importance and merit, it is indubitably essential to test the performance of VLMs and foundation models such as SAM, on these images. In this study, we charge ChatGPT, LLaVA, Gemini, and SAM with classification, segmentation, counting, and VQA tasks on a variety of microscopy images. We observe that ChatGPT and Gemini are impressively able to comprehend the visual features in microscopy images, while SAM is quite capable at isolating artefacts in a general sense. However, the performance is not close to that of a domain expert - the models are readily encumbered by the introduction of impurities, defects, artefact overlaps and diversity present in the images.
Abstract (translated)
近年来，随着Vision语言模型（VLMs）的出现，它们在理解图像和文本数据的双模态方面得到了关注。例如，LLaVA、ChatGPT-4和Gemini等VLM最近在自然图像描述性、视觉问答（VQA）和空间推理等任务中表现出色。此外，由元人工智能（Meta AI）开发的普遍分割模型Semantic Anywhere Model（SAM）在从未见过的图像中隔离物体方面表现出史无前例的性能。由于医疗专家、生物学家和材料科学家通常将显微镜图像或医学图像与文本信息（标题、文献或报告）一起检查，并从中得出重要且有益的结论，因此测试VLM和基础模型（如SAM）在这些图像上的性能无疑至关重要。在这项研究中，我们对ChatGPT、LLaVA、Gemini和SAM在各种显微镜图像上执行分类、分割、计数和VQA任务。我们观察到，ChatGPT和Gemini在显微镜图像的视觉特征方面表现出惊人的理解能力，而SAM在分离总体上的伪影方面表现相当出色。然而，这些模型的性能与领域专家的相当距离，模型很容易受到图像中存在的杂质、缺陷、伪影和多样性等因素的影响。
URL
https://arxiv.org/abs/2405.00876
