Abstract
Multi-modal retrieval-augmented generation (MM-RAG) promises grounded biomedical QA, but it is unclear when to (i) convert figures/tables into text versus (ii) use optical character recognition (OCR)-free visual retrieval that returns page images and leaves interpretation to the generator. We study this trade-off in glycobiology, a visually dense domain. We built a benchmark of 120 multiple-choice questions (MCQs) from 25 papers, stratified by retrieval difficulty (easy text, medium figures/tables, hard cross-evidence). We implemented four augmentations-None, Text RAG, Multi-modal conversion, and late-interaction visual retrieval (ColPali)-using Docling parsing and Qdrant indexing. We evaluated mid-size open-source and frontier proprietary models (e.g., Gemma-3-27B-IT, GPT-4o family). Additional testing used the GPT-5 family and multiple visual retrievers (ColPali/ColQwen/ColFlor). Accuracy with Agresti-Coull 95% confidence intervals (CIs) was computed over 5 runs per configuration. With Gemma-3-27B-IT, Text and Multi-modal augmentation outperformed OCR-free retrieval (0.722-0.740 vs. 0.510 average accuracy). With GPT-4o, Multi-modal achieved 0.808, with Text 0.782 and ColPali 0.745 close behind; within-model differences were small. In follow-on experiments with the GPT-5 family, the best results with ColPali and ColFlor improved by ~2% to 0.828 in both cases. In general, across the GPT-5 family, ColPali, ColQwen, and ColFlor were statistically indistinguishable. GPT-5-nano trailed larger GPT-5 variants by roughly 8-10%. Pipeline choice is capacity-dependent: converting visuals to text lowers the reader burden and is more reliable for mid-size models, whereas OCR-free visual retrieval becomes competitive under frontier models. Among retrievers, ColFlor offers parity with heavier options at a smaller footprint, making it an efficient default when strong generators are available.
Abstract (translated)
多模态检索增强生成（MM-RAG）为生物医学问答提供了坚实的基础，但何时将图表转换成文本或使用无需光学字符识别（OCR）的视觉检索来返回页面图像并让生成器进行解释尚不清楚。我们在糖生物学这一视觉密集型领域研究了这种权衡。我们建立了一个包含120个多项选择题（MCQs）的基准测试，这些题目来自25篇论文，并根据检索难度分为三类：简单文本、中等图表/表格和困难跨证据问题。我们实现了四种增强方法——无增强、Text RAG、多模态转换以及迟延交互视觉检索（ColPali），使用Docling解析与Qdrant索引技术。我们评估了中型开源模型和前沿专有模型（例如Gemma-3-27B-IT，GPT-4o家族）。额外的测试使用了GPT-5家族及多种视觉检索器（ColPali/ColQwen/ColFlor）。通过五次运行每个配置后的Agresti-Coull 95%置信区间计算准确率。在使用Gemma-3-27B-IT时，Text和多模态增强优于无OCR的视觉检索（0.722至0.740对0.510平均精度）。对于GPT-4o模型，多模态方法达到了0.808的准确率，而文本模式为0.782，ColPali则略低至0.745；同一模型内部差异较小。在后续使用GPT-5家族的实验中，ColPali和ColFlor的最佳结果提升了大约2%达到0.828（两个案例均是）。总的来说，在整个GPT-5家族中，ColPali、ColQwen和ColFlor统计上没有显著差异。而GPT-5-nano则落后于较大的GPT-5变体约8%-10%。 管道选择取决于容量：将视觉元素转换为文本降低了阅读负担，并且对于中型模型来说更为可靠，而对于前沿模型而言，则无需OCR的视觉检索更具竞争力。在检索器方面，ColFlor提供了轻量级的解决方案，其性能与较重的选择相媲美，在强生成器可用时成为高效默认选择。
URL
https://arxiv.org/abs/2512.16802
