Abstract
Video Multimodal Large Language Models (VideoMLLMs) have achieved remarkable progress in both Video-to-Text and Text-to-Video tasks. However, they often suffer fro hallucinations, generating content that contradicts the visual input. Existing evaluation methods are limited to one task (e.g., V2T) and also fail to assess hallucinations in open-ended, free-form responses. To address this gap, we propose FIFA, a unified FaIthFulness evAluation framework that extracts comprehensive descriptive facts, models their semantic dependencies via a Spatio-Temporal Semantic Dependency Graph, and verifies them using VideoQA models. We further introduce Post-Correction, a tool-based correction framework that revises hallucinated content. Extensive experiments demonstrate that FIFA aligns more closely with human judgment than existing evaluation methods, and that Post-Correction effectively improves factual consistency in both text and video generation.
Abstract (translated)
视频多模态大型语言模型（VideoMLLM）在视频到文本和文本到视频任务中取得了显著进展。然而，这些模型常常出现幻觉问题，即生成的内容与视觉输入相矛盾。现有的评估方法仅限于单一任务（如V2T），并且无法对开放性、自由形式回复中的幻觉进行评估。为了填补这一空白，我们提出了FIFA框架，这是一个统一的忠实度评估框架，能够提取全面描述性的事实，并通过时空语义依赖图模型这些事实之间的语义关系，并利用视频问答模型验证它们的真实性和一致性。此外，我们还引入了Post-Correction工具，这是一种基于工具的纠正框架，用于修订幻觉内容。 广泛的实验表明，FIFA在与人类判断的一致性上优于现有的评估方法，并且Post-Correction有效地提高了文本和视频生成中的事实一致性和准确性。
URL
https://arxiv.org/abs/2507.06523
