Abstract
In real-world scenarios, providing user queries with visually enhanced responses can considerably benefit understanding and memory, underscoring the great value of interleaved image-text generation. Despite recent progress, like the visual autoregressive model that unifies text and image processing in a single transformer architecture, generating high-quality interleaved content remains challenging. Moreover, evaluations of these interleaved sequences largely remain underexplored, with existing benchmarks often limited by unimodal metrics that inadequately assess the intricacies of combined image-text outputs. To address these issues, we present RAG-IGBench, a thorough benchmark designed specifically to evaluate the task of Interleaved Generation based on Retrieval-Augmented Generation (RAG-IG) in open-domain question answering. RAG-IG integrates multimodal large language models (MLLMs) with retrieval mechanisms, enabling the models to access external image-text information for generating coherent multimodal content. Distinct from previous datasets, RAG-IGBench draws on the latest publicly available content from social platforms and introduces innovative evaluation metrics that measure the quality of text and images, as well as their consistency. Through extensive experiments with state-of-the-art MLLMs (both open-source and proprietary) on RAG-IGBench, we provide an in-depth analysis examining the capabilities and limitations of these models. Additionally, we validate our evaluation metrics by demonstrating their high correlation with human assessments. Models fine-tuned on RAG-IGBench's training set exhibit improved performance across multiple benchmarks, confirming both the quality and practical utility of our dataset. Our benchmark is available at this https URL.
Abstract (translated)
在实际应用场景中，通过视觉增强的响应来提供用户查询可以显著提高理解和记忆的效果，这突显了交错式图像-文本生成的巨大价值。尽管最近取得了进展，例如将文字和图像处理统一在一个Transformer架构中的视觉自回归模型，但要生成高质量的交错式内容仍然具有挑战性。此外，对这些交错序列的评估仍很大程度上被忽略，现有的基准测试往往受限于单模态度量标准，无法充分评价组合式图像-文本输出的复杂性。 为解决这些问题，我们提出了RAG-IGBench，这是一个专门针对基于检索增强生成（Retrieval-Augmented Generation, RAG）任务进行交错式生成评估的全面基准。RAG-IG结合了多模态大型语言模型（Multimodal Large Language Models, MLLMs）与检索机制，使这些模型能够访问外部图像文本信息以生成连贯的多模态内容。不同于之前的语料库，RAG-IGBench借鉴了最新的公开社交平台上的内容，并引入了一系列创新评估指标来衡量文本和图片的质量及其一致性。 通过对最先进的MLLMs（包括开源与专有）进行广泛的实验测试，并在RAG-IGBench上提供深入分析，我们揭示了这些模型的能力及局限性。此外，通过证明我们的评价标准与人工判断的高度相关性，进一步验证了我们的评估指标的有效性。基于RAG-IGBench训练集微调的模型，在多个基准测试中的表现都有所提升，这证实了我们数据集的质量和实用性。 我们的基准可以在以下网址获得：[请在此处插入链接]。
URL
https://arxiv.org/abs/2512.05119
