Abstract
Camera-controlled generative video re-rendering methods, such as ReCamMaster, have achieved remarkable progress. However, despite their success in single-view setting, these works often struggle to maintain consistency across multi-view scenarios. Ensuring spatio-temporal coherence in hallucinated regions remains challenging due to the inherent stochasticity of generative models. To address it, we introduce PlenopticDreamer, a framework that synchronizes generative hallucinations to maintain spatio-temporal memory. The core idea is to train a multi-in-single-out video-conditioned model in an autoregressive manner, aided by a camera-guided video retrieval strategy that adaptively selects salient videos from previous generations as conditional inputs. In addition, Our training incorporates progressive context-scaling to improve convergence, self-conditioning to enhance robustness against long-range visual degradation caused by error accumulation, and a long-video conditioning mechanism to support extended video generation. Extensive experiments on the Basic and Agibot benchmarks demonstrate that PlenopticDreamer achieves state-of-the-art video re-rendering, delivering superior view synchronization, high-fidelity visuals, accurate camera control, and diverse view transformations (e.g., third-person to third-person, and head-view to gripper-view in robotic manipulation). Project page: this https URL
Abstract (translated)
相机控制的生成视频重渲染方法，如ReCamMaster，已经取得了显著的进步。然而，在单一视角设置下尽管这些方法表现出色，但在多视角场景中却往往难以保持一致性。由于生成模型固有的随机性，确保幻觉区域的空间和时间连贯性仍然是一项挑战。为了解决这个问题，我们提出了PlenopticDreamer框架，该框架通过同步生成的幻象来维持空间和时间记忆的一致性。核心思想是训练一个以多输入单输出视频条件模型自回归方式进行训练，并辅以相机引导的视频检索策略，这种策略可以动态地从先前的生成中选择显著视频作为条件输入。此外，在训练过程中还加入了渐进式上下文扩展来提高收敛速度，自我调节来增强对长时间视觉退化（由错误累积引起）的鲁棒性，以及支持长视频生成的长期视频条件机制。在Basic和Agibot基准测试上的广泛实验表明，PlenopticDreamer实现了最先进的视频重渲染性能，在视角同步、高保真视觉效果、准确的相机控制以及多样化的视图转换（如第三人称到第三人称，或机器人操作中的头部视图到机械臂视图）等方面表现出色。项目页面：[此链接](https://this-url.com/)
URL
https://arxiv.org/abs/2601.05239
