Abstract
We introduce a recipe for generating immersive 3D worlds from a single image by framing the task as an in-context learning problem for 2D inpainting models. This approach requires minimal training and uses existing generative models. Our process involves two steps: generating coherent panoramas using a pre-trained diffusion model and lifting these into 3D with a metric depth estimator. We then fill unobserved regions by conditioning the inpainting model on rendered point clouds, requiring minimal fine-tuning. Tested on both synthetic and real images, our method produces high-quality 3D environments suitable for VR display. By explicitly modeling the 3D structure of the generated environment from the start, our approach consistently outperforms state-of-the-art, video synthesis-based methods along multiple quantitative image quality metrics. Project Page: this https URL
Abstract (translated)
我们介绍了一种从单张图像生成沉浸式三维世界的食谱，将此任务视为2D图像修复模型的在上下文学习问题。这种方法需要最少的训练，并利用现有的生成模型。我们的流程包括两个步骤：使用预训练的扩散模型生成连贯的全景图，然后通过度量深度估计器将其提升到3D空间中。接着，我们通过对渲染的点云进行条件设置来填充未观察到的区域，这只需要少量微调即可完成。我们在合成图像和真实图像上进行了测试，我们的方法能够产生高质量的三维环境，适合用于虚拟现实显示。 通过从一开始就明确模拟生成环境中三维结构的特点，我们的方法在多个定量图像质量指标上始终优于基于视频合成的方法。项目页面：[此链接](https://this https URL)
URL
https://arxiv.org/abs/2503.16611
