Abstract
In this paper, we propose Extend3D, a training-free pipeline for 3D scene generation from a single image, built upon an object-centric 3D generative model. To overcome the limitations of fixed-size latent spaces in object-centric models for representing wide scenes, we extend the latent space in the $x$ and $y$ directions. Then, by dividing the extended latent space into overlapping patches, we apply the object-centric 3D generative model to each patch and couple them at each time step. Since patch-wise 3D generation with image conditioning requires strict spatial alignment between image and latent patches, we initialize the scene using a point cloud prior from a monocular depth estimator and iteratively refine occluded regions through SDEdit. We discovered that treating the incompleteness of 3D structure as noise during 3D refinement enables 3D completion via a concept, which we term under-noising. Furthermore, to address the sub-optimality of object-centric models for sub-scene generation, we optimize the extended latent during denoising, ensuring that the denoising trajectories remain consistent with the sub-scene dynamics. To this end, we introduce 3D-aware optimization objectives for improved geometric structure and texture fidelity. We demonstrate that our method yields better results than prior methods, as evidenced by human preference and quantitative experiments.
Abstract (translated)
本文提出Extend3D，一种基于以物体为中心的3D生成模型、无需训练的从单张图像生成3D场景的流程。为克服以物体为中心的模型在表示广阔场景时潜在空间尺寸固定的局限，我们在$x$和$y$方向上扩展潜在空间。随后，通过将扩展后的潜在空间划分为重叠的图块，我们将以物体为中心的3D生成模型应用于每个图块，并在每个时间步进行耦合。由于带图像条件的分块3D生成需要图像与潜在图块间严格的空间对齐，我们利用单目深度估计器的点云先验初始化场景，并通过SDEdit迭代优化遮挡区域。我们发现，在3D细化过程中将3D结构的不完整性视为噪声，能够通过一种我们称为“欠去噪”的概念实现3D补全。此外，为解决以物体为中心的模型在子场景生成上的次优性，我们在去噪过程中优化扩展潜在空间，确保去噪轨迹与子场景动态保持一致。为此，我们引入了3D感知的优化目标，以提升几何结构与纹理保真度。实验表明，我们的方法在人类偏好评估与定量实验中均优于先前方法。
URL
https://arxiv.org/abs/2603.29387
