Abstract
Existing vision language models (VLMs), including GPT-4 and DALL-E, often struggle to preserve logic, object identity, and style in multimodal image-text generation. This limitation significantly hinders the generalization capability of VLMs in complex image-text input-output scenarios. To address this issue, we propose IUT-Plug, a module grounded in an Image Understanding Tree (IUT), which enhances existing interleaved VLMs through explicit structured reasoning, thereby mitigating context drift in logic, entity identity, and style. The proposed framework operates in two stages. (1) A dynamic IUT-Plug extraction module parses visual scenes into hierarchical symbolic structures. (2) A coordinated narrative-flow and image synthesis mechanism ensures cross-modal consistency. To evaluate our approach, we construct a novel benchmark based on 3,000 real human-generated question-answer pairs over fine-tuned large models, introducing a dynamic evaluation protocol for quantifying context drift in interleaved VLMs. Experimental results demonstrate that IUT-Plug not only improves accuracy on established benchmarks but also effectively alleviates the three critical forms of context drift across diverse multimodal question answering (QA) scenarios.
Abstract (translated)
现有的视觉语言模型（VLM），包括GPT-4和DALL-E，在多模态图像文本生成中通常难以保持逻辑、对象身份和风格的一致性。这一局限极大地阻碍了VLM在复杂图像文字输入输出场景中的泛化能力。为了解决这个问题，我们提出了基于图像理解树（IUT）的IUT-Plug模块，通过显式的结构化推理来增强现有的交错式视觉语言模型（interleaved VLM），从而减轻逻辑、实体身份和风格方面的上下文漂移问题。 该框架分为两个阶段操作： 1. 动态IUT-Plug提取模块将视觉场景解析为分层的符号结构。 2. 协同叙事流程与图像合成机制确保跨模态的一致性。 为了评估我们的方法，我们构建了一个基于3,000对真实的人类生成的问题答案对的新基准测试，并引入了一种量化交错式VLM中上下文漂移的动态评价协议。实验结果表明，IUT-Plug不仅在现有的基准上提高了准确性，而且还有效地缓解了跨多种多模态问答场景中的三种关键形式的上下文漂移问题。
URL
https://arxiv.org/abs/2510.10969
