Abstract
Foundational world models must be both interactive and preserve spatiotemporal coherence for effective future planning with action choices. However, present models for long video generation have limited inherent world modeling capabilities due to two main challenges: compounding errors and insufficient memory mechanisms. We enhance image-to-video models with interactive capabilities through additional action conditioning and autoregressive framework, and reveal that compounding error is inherently irreducible in autoregressive video generation, while insufficient memory mechanism leads to incoherence of world models. We propose video retrieval augmented generation (VRAG) with explicit global state conditioning, which significantly reduces long-term compounding errors and increases spatiotemporal consistency of world models. In contrast, naive autoregressive generation with extended context windows and retrieval-augmented generation prove less effective for video generation, primarily due to the limited in-context learning capabilities of current video models. Our work illuminates the fundamental challenges in video world models and establishes a comprehensive benchmark for improving video generation models with internal world modeling capabilities.
Abstract (translated)
基础世界模型必须既具有交互性又能够保持时空一致性，以便在有效规划未来行动时做出选择。然而，目前用于长视频生成的模型由于两个主要挑战而具备有限的世界建模能力：累积误差和不足的记忆机制。我们通过添加行为条件化和自回归框架来增强图像到视频模型的互动功能，并揭示了自回归视频生成中的累积错误本质上是不可减少的，而缺乏记忆机制则会导致世界模型不一致。为此，我们提出了一种基于视频检索增强生成（VRAG）的方法，该方法通过明确的全局状态条件化显著减少了长期累积误差并增加了时空一致性。 相比之下，简单的具有扩展上下文窗口和检索增强的自回归生成对于视频生成效果较差，主要是因为目前的视频模型在上下文学习能力方面存在限制。我们的工作揭示了视频世界模型中的基本挑战，并为改进具备内部世界建模功能的视频生成模型建立了全面基准。
URL
https://arxiv.org/abs/2505.21996
