Abstract
Recent advances in large reconstruction and generative models have significantly improved scene reconstruction and novel view generation. However, due to compute limitations, each inference with these large models is confined to a small area, making long-range consistent scene generation challenging. To address this, we propose StarGen, a novel framework that employs a pre-trained video diffusion model in an autoregressive manner for long-range scene generation. The generation of each video clip is conditioned on the 3D warping of spatially adjacent images and the temporally overlapping image from previously generated clips, improving spatiotemporal consistency in long-range scene generation with precise pose control. The spatiotemporal condition is compatible with various input conditions, facilitating diverse tasks, including sparse view interpolation, perpetual view generation, and layout-conditioned city generation. Quantitative and qualitative evaluations demonstrate StarGen's superior scalability, fidelity, and pose accuracy compared to state-of-the-art methods.
Abstract (translated)
最近在大规模重建和生成模型方面的进展显著提升了场景重建和新颖视图生成的性能。然而，由于计算限制，这些大型模型每次推理只能处理一个小区域，这使得长距离的一致性场景生成变得具有挑战性。为了解决这个问题，我们提出了StarGen，这是一个新的框架，它采用预训练的视频扩散模型以自回归方式用于长距离场景生成。每个视频片段的生成受到空间相邻图像和之前生成片段中时间上重叠的图像的空间扭曲条件的影响，这提高了长时间范围内场景生成中的时空一致性，并实现了精确的姿态控制。这种时空条件可以与各种输入条件兼容，从而能够支持多种任务，包括稀疏视图插值、持续视图生成以及布局条件下的城市生成。定量和定性的评估表明，StarGen在可扩展性、保真度和姿态准确性方面优于现有的方法。
URL
https://arxiv.org/abs/2501.05763
