Abstract
In recent years, 3D generation has made great strides in both academia and industry. However, generating 3D scenes from a single RGB image remains a significant challenge, as current approaches often struggle to ensure both object generation quality and scene coherence in multi-object scenarios. To overcome these limitations, we propose a novel three-stage framework for 3D scene generation with explicit geometric representations and high-quality textural details via single image-guided model generation and spatial layout optimization. Our method begins with an image instance segmentation and inpainting phase, which recovers missing details of occluded objects in the input images, thereby achieving complete generation of foreground 3D assets. Subsequently, our approach captures the spatial geometry of reference image by constructing pseudo-stereo viewpoint for camera parameter estimation and scene depth inference, while employing a model selection strategy to ensure optimal alignment between the 3D assets generated in the previous step and the input. Finally, through model parameterization and minimization of the Chamfer distance between point clouds in 3D and 2D space, our approach optimizes layout parameters to produce an explicit 3D scene representation that maintains precise alignment with input guidance image. Extensive experiments on multi-object scene image sets have demonstrated that our approach not only outperforms state-of-the-art methods in terms of geometric accuracy and texture fidelity of individual generated 3D models, but also has significant advantages in scene layout synthesis.
Abstract (translated)
近年来，3D生成技术在学术界和工业界都取得了重大进展。然而，从单张RGB图像生成3D场景仍然是一个巨大的挑战，因为现有方法常常难以同时保证多对象场景中的物体生成质量和场景一致性。为了克服这些限制，我们提出了一种新颖的三阶段框架，用于通过单一图像引导模型生成和空间布局优化来创建具有显式几何表示和高质量纹理细节的3D场景。 我们的方法首先进行图像实例分割与修复阶段，在此过程中恢复输入图片中被遮挡物体缺失的详细信息，从而实现前景3D资产的完整生成。随后，我们通过构建伪立体视角来进行相机参数估计和场景深度推断，以此捕捉参考图的空间几何结构，并采用模型选择策略确保先前步骤产生的3D资产与输入图像的最佳对齐。最后，通过对模型进行参数化处理并最小化3D和2D空间中点云之间的Chamfer距离，我们的方法优化布局参数，从而生成一个显式的3D场景表示，保持精确地与指导输入图一致。 在多对象场景图像集上的广泛实验表明，我们所提出的方法不仅在单个生成的3D模型的几何准确性和纹理保真度方面超越了最先进的技术，还在场景布局合成方面具有显著优势。
URL
https://arxiv.org/abs/2507.14841
