Abstract
Vision-Language Models (VLMs) now generate discourse-level, multi-sentence visual descriptions, challenging text scene graph parsers originally designed for single-sentence caption-to-graph mapping. Current approaches typically merge sentence-level parsing outputs for discourse input, often missing phenomena like cross-sentence coreference, resulting in fragmented graphs and degraded downstream VLM task performance. To address this, we introduce a new task, Discourse-level text Scene Graph parsing (DiscoSG), supported by our dataset DiscoSG-DS, which comprises 400 expert-annotated and 8,430 synthesised multi-sentence caption-graph pairs for images. Each caption averages 9 sentences, and each graph contains at least 3 times more triples than those in existing datasets. While fine-tuning large PLMs (i.e., GPT-4) on DiscoSG-DS improves SPICE by approximately 48% over the best sentence-merging baseline, high inference cost and restrictive licensing hinder its open-source use, and smaller fine-tuned PLMs struggle with complex graphs. We propose DiscoSG-Refiner, which drafts a base graph using one small PLM, then employs a second PLM to iteratively propose graph edits, reducing full-graph generation overhead. Using two Flan-T5-Base models, DiscoSG-Refiner still improves SPICE by approximately 30% over the best baseline while achieving 86 times faster inference than GPT-4. It also consistently improves downstream VLM tasks like discourse-level caption evaluation and hallucination detection. Code and data are available at: this https URL
Abstract (translated)
视觉语言模型（VLMs）现在生成的是以对话层面、多句式描述为主的视觉描述，这挑战了最初为单句描述到场景图映射设计的文本场景图解析器。当前的方法通常通过合并句子级别的解析输出来处理对话输入，但这种方式常常遗漏跨句指代等现象，导致产生的图碎片化，并且影响下游VLM任务的表现。 为了应对这一挑战，我们引入了一个新的任务——话语级文本场景图解析（DiscoSG），并为此构建了数据集DiscoSG-DS。该数据集包括400个专家注释的和8,430对合成的多句描述与对应场景图配对的数据。每个描述平均包含9句话，而每个图表至少包含了现有数据集中三倍以上的三元组数量。 虽然在DiscoSG-DS上微调大型PLM（如GPT-4）可以使SPICE评分比最佳句子合并基线提高约48%，但由于高昂的推理成本和限制性许可条款阻碍了其开源使用，而且较小规模的微调模型难以处理复杂的图。我们提出了一种名为DiscoSG-Refiner的方法：首先利用一个小一点的语言模型生成基础图；然后采用第二个语言模型进行迭代式的图编辑建议，从而减少整个图表生成的工作量。 通过使用两个Flan-T5-Base模型，DiscoSG-Refiner仍能比最佳基线提高约30%的SPICE评分，并且推理速度比GPT-4快86倍。此外，在对话级描述评估和幻觉检测等下游VLM任务上也表现出稳定地提升性能。 代码与数据可在以下网址获取：[请提供具体的URL链接]
URL
https://arxiv.org/abs/2506.15583
