Abstract
Foundation models increasingly offer potential to support interactive, agentic workflows that assist researchers during analysis and interpretation of image data. Such workflows often require coupling vision to language to provide a natural-language interface. However, paired image-text data needed to learn this coupling are scarce and difficult to obtain in many research and clinical settings. One such setting is microscopic analysis of cell-body-stained histological human brain sections, which enables the study of cytoarchitecture: cell density and morphology and their laminar and areal organization. Here, we propose a label-mediated method that generates meaningful captions from images by linking images and text only through a label, without requiring curated paired image-text data. Given the label, we automatically mine area descriptions from related literature and use them as synthetic captions reflecting canonical cytoarchitectonic attributes. An existing cytoarchitectonic vision foundation model (CytoNet) is then coupled to a large language model via an image-to-text training objective, enabling microscopy regions to be described in natural language. Across 57 brain areas, the resulting method produces plausible area-level descriptions and supports open-set use through explicit rejection of unseen areas. It matches the cytoarchitectonic reference label for in-scope patches with 90.6% accuracy and, with the area label masked, its descriptions remain discriminative enough to recover the area in an 8-way test with 68.6% accuracy. These results suggest that weak, label-mediated pairing can suffice to connect existing biomedical vision foundation models to language, providing a practical recipe for integrating natural-language in domains where fine-grained paired annotations are scarce.
Abstract (translated)
基础模型日益展现出支持互动、代理工作流程的潜力，这些工作流程能在图像数据分析过程中协助研究人员。这类工作流程通常需要将视觉与语言相结合，以提供自然语言界面。然而，在许多研究和临床环境中获取学习这种结合所需的配对图像文本数据却非常稀缺且难以获得。一个典型的例子是人类大脑切片中细胞体着色组织学的显微分析，这有助于研究细胞结构：包括细胞密度、形态以及它们在层状和区域中的排列方式。在此背景下，我们提出了一种标签介导的方法，该方法通过标签将图像与文本链接起来生成有意义的描述，而无需依赖精心策划的配对图像文本数据。给定特定标签后，我们可以自动从相关文献中挖掘出对应区域的描述，并将其用作合成说明，反映规范化的细胞结构属性。然后，现有的细胞结构视觉基础模型（CytoNet）通过一个将图像转换为文本的训练目标与大型语言模型相结合，使得显微镜下观察到的区域能够以自然语言进行描述。在涉及57个脑区的情况下，该方法产生的描述符具备了合理的区域性特征，并支持开放集使用，通过对未见过的区域明确拒绝来实现这一点。对于在范围内的补丁，其匹配细胞结构参考标签准确率为90.6%，并且即使遮蔽了区域标签，在一个八分类测试中仍能以68.6%的准确率恢复出该区域。 这些结果表明，弱标签介导配对足以将现有的生物医学视觉基础模型与语言连接起来，在精细标注稀缺的情况下为自然语言的集成提供了一个实际的方法。
URL
https://arxiv.org/abs/2602.23088
