Abstract
Multimodal large language models (MLLMs) have achieved remarkable progress in video understanding. However, seemingly plausible outputs often suffer from poor visual and temporal grounding: a model may fabricate object existence, assign incorrect attributes, or collapse repeated events while still producing a globally reasonable caption or answer. We study this failure mode through a compositional consistency audit that decomposes a caption into supporting factual and temporal claims, investigating whether a correct high-level prediction is actually backed by valid lower-level evidence. Our top-down audit reveals that even correct root relational claims often lack reliable attribute and existence support. This indicates that standard sentence-level supervision is a weak proxy for faithful video understanding. Furthermore, when turning to reinforcement learning (RL) for better alignment, standard sentence-level rewards often prove too coarse to accurately localize specific grounding failures. To address this, we replace generic sentence-level rewards with a structured reward built from factual and temporal units. Our training objective integrates three complementary components: (1) an instance-aware scene-graph reward for factual objects, attributes, and relations; (2) a temporal reward for event ordering and repetition; and (3) a video-grounded VQA reward for hierarchical self-verification. Across temporal, general video understanding, and hallucination-oriented benchmarks, this objective yields consistent gains on open-source backbones. These results suggest that structured reward shaping is a practical route to more faithful video understanding.
Abstract (translated)
多模态大语言模型（MLLMs）在视频理解领域已取得显著进展。然而，看似合理的输出往往存在视觉与时间定位不准确的问题：模型可能虚构物体存在、赋予错误属性，或在重复事件处理上出现偏差，却仍生成整体通顺的说明或回答。本研究通过组合一致性审计方法，将说明分解为支撑性事实主张与时间主张，探究高层预测是否真正具备有效的底层证据支持。这种自上而下的审计揭示，即使正确的关系性主张也常缺乏可靠的属性与存在性支撑，表明标准句子级监督对忠实视频理解而言是较弱的代理指标。此外，当转向强化学习（RL）以优化对齐时，标准句子级奖励往往过于粗糙，难以精确定位具体定位失败。为此，我们采用基于事实与时间单元的结构化奖励替代通用句子级奖励。训练目标整合三个互补组件：（1）面向实例的场景图奖励，用于事实性对象、属性与关系；（2）针对事件排序与重复的时间奖励；（3）用于分层自验证的视频基础VQA奖励。在时间理解、通用视频理解及面向幻觉的基准测试中，该目标在开源模型上均实现稳定提升。结果表明，结构化奖励塑造是提升视频理解忠实度的可行路径。
URL
https://arxiv.org/abs/2604.01460
