Abstract
Autoregressive video diffusion models have proved effective for world modeling and interactive scene generation, with Minecraft gameplay as a representative application. To faithfully simulate play, a model must generate natural content while exploring new scenes and preserve spatial consistency when revisiting explored areas. Under limited computation budgets, it must compress and exploit historical cues within a finite context window, which exposes a trade-off: Temporal-only memory lacks long-term spatial consistency, whereas adding spatial memory strengthens consistency but may degrade new scene generation quality when the model over-relies on insufficient spatial context. We present Memory Forcing, a learning framework that pairs training protocols with a geometry-indexed spatial memory. Hybrid Training exposes distinct gameplay regimes, guiding the model to rely on temporal memory during exploration and incorporate spatial memory for revisits. Chained Forward Training extends autoregressive training with model rollouts, where chained predictions create larger pose variations and encourage reliance on spatial memory for maintaining consistency. Point-to-Frame Retrieval efficiently retrieves history by mapping currently visible points to their source frames, while Incremental 3D Reconstruction maintains and updates an explicit 3D cache. Extensive experiments demonstrate that Memory Forcing achieves superior long-term spatial consistency and generative quality across diverse environments, while maintaining computational efficiency for extended sequences.
Abstract (translated)
自回归视频扩散模型已被证明在世界建模和交互式场景生成方面非常有效，以《我的世界》游戏玩法作为代表性应用。为了真实地模拟游戏体验，一个模型必须在探索新场景时生成自然内容，并且当重新访问已探索的区域时保持空间一致性。在计算资源有限的情况下，该模型需要压缩并在有限的时间窗口内利用历史线索，这暴露了一个权衡：仅时间记忆缺乏长期的空间一致性，而添加空间记忆虽然可以增强一致性，但可能会因过度依赖不足的空间上下文而导致新场景生成质量下降。 我们提出了Memory Forcing（内存强制）框架，这是一个结合了几何索引空间内存的训练协议的学习框架。混合训练通过区分不同的游戏模式，引导模型在探索阶段依赖时间记忆，并在重访时引入空间记忆以保持一致性。链式前向训练则是在自回归训练的基础上添加了模型预测序列，在这种情况下，连续的预测会创建更大的姿态变化，并鼓励模型依赖空间内存来维持一致性。点到帧检索通过将当前可见点映射回其源帧来进行历史记录的有效检索，而增量3D重建则维护并更新显式的三维缓存。 广泛的实验表明，Memory Forcing框架在各种环境中实现了卓越的长期空间一致性和生成质量，同时保持了对延长序列计算效率的需求。
URL
https://arxiv.org/abs/2510.03198
