Abstract
Visual storytelling systems struggle to maintain character identity across frames and link actions to appropriate subjects, frequently leading to referential hallucinations. These issues can be addressed through grounding of characters, objects, and other entities on the visual elements. We propose StoryReasoning, a dataset containing 4,178 stories derived from 52,016 movie images, with both structured scene analyses and grounded stories. Each story maintains character and object consistency across frames while explicitly modeling multi-frame relationships through structured tabular representations. Our approach features cross-frame object re-identification using visual similarity and face recognition, chain-of-thought reasoning for explicit narrative modeling, and a grounding scheme that links textual elements to visual entities across multiple frames. We establish baseline performance by fine-tuning Qwen2.5-VL 7B, creating Qwen Storyteller, which performs end-to-end object detection, re-identification, and landmark detection while maintaining consistent object references throughout the story. Evaluation demonstrates a reduction from 4.06 to 3.56 (-12.3%) hallucinations on average per story when compared to a non-fine-tuned model.
Abstract (translated)
视觉叙事系统在维护角色身份以及将动作与适当主体关联方面面临挑战，这常常导致指代幻觉。这些问题可以通过基于视觉元素对人物、物体和其他实体进行定位来解决。我们提出了StoryReasoning数据集，包含从52,016张电影图片中提取的4,178个故事，并提供了结构化的场景分析和基于视觉定位的故事文本。每个故事在不同帧之间保持了角色和对象的一致性，并通过结构化表格表示显式地建模多帧之间的关系。 我们的方法包括利用视觉相似度和面部识别进行跨帧对象重新识别，链式思维推理用于明确叙述模型构建，以及将文本元素链接到多个帧中的视觉实体的定位方案。我们通过对Qwen2.5-VL 7B进行微调来建立基线性能，创建了Qwen讲故事模型，该模型在故事中保持一致的对象引用的同时执行端到端的对象检测、重新识别和地标检测。评估结果表明，与未经微调的模型相比，平均每个故事中的幻觉减少了12.3%（从4.06减少到3.56）。
URL
https://arxiv.org/abs/2505.10292
