Abstract
We propose a novel memory mechanism to build video generators that can explore environments interactively. Similar results have previously been achieved by out-painting 2D views of the scene while incrementally reconstructing its 3D geometry, which quickly accumulates errors, or by video generators with a short context window, which struggle to maintain scene coherence over the long term. To address these limitations, we introduce Surfel-Indexed View Memory (VMem), a mechanism that remembers past views by indexing them geometrically based on the 3D surface elements (surfels) they have observed. VMem enables the efficient retrieval of the most relevant past views when generating new ones. By focusing only on these relevant views, our method produces consistent explorations of imagined environments at a fraction of the computational cost of using all past views as context. We evaluate our approach on challenging long-term scene synthesis benchmarks and demonstrate superior performance compared to existing methods in maintaining scene coherence and camera control.
Abstract (translated)
我们提出了一种新颖的记忆机制，用于构建能够进行交互式环境探索的视频生成器。此前曾通过扩展场景的二维视图并逐步重建其三维几何结构来实现类似结果，但这种方法会迅速积累误差，或者使用短上下文窗口的视频生成器难以在长期内保持场景的一致性。为了解决这些限制，我们引入了基于3D表面元素（surfels）几何索引过去视图的视图记忆机制（VMem）。通过这种方式，当生成新视图时，可以高效地检索最相关的过去视图。我们的方法仅关注这些相关视图，在保持计算成本较低的同时，能够产生连贯性较强的想象环境探索结果。 我们在具有挑战性的长期场景合成基准测试中评估了这种方法，并展示了在维持场景一致性和相机控制方面优于现有方法的性能。
URL
https://arxiv.org/abs/2506.18903
