Abstract
In the field of 3D content generation, single image scene reconstruction methods still struggle to simultaneously ensure the quality of individual assets and the coherence of the overall scene in complex environments, while texture editing techniques often fail to maintain both local continuity and multi-view consistency. In this paper, we propose a novel system ZeroScene, which leverages the prior knowledge of large vision models to accomplish both single image-to-3D scene reconstruction and texture editing in a zero-shot manner. ZeroScene extracts object-level 2D segmentation and depth information from input images to infer spatial relationships within the scene. It then jointly optimizes 3D and 2D projection losses of the point cloud to update object poses for precise scene alignment, ultimately constructing a coherent and complete 3D scene that encompasses both foreground and background. Moreover, ZeroScene supports texture editing of objects in the scene. By imposing constraints on the diffusion model and introducing a mask-guided progressive image generation strategy, we effectively maintain texture consistency across multiple viewpoints and further enhance the realism of rendered results through Physically Based Rendering (PBR) material estimation. Experimental results demonstrate that our framework not only ensures the geometric and appearance accuracy of generated assets, but also faithfully reconstructs scene layouts and produces highly detailed textures that closely align with text prompts.
Abstract (translated)
在3D内容生成领域，单张图像场景重建方法仍难以同时保证复杂环境中各个资产的质量和整个场景的连贯性。此外，纹理编辑技术往往无法兼顾局部连续性和多视角一致性。为此，本文提出了一种新的系统ZeroScene，该系统利用大型视觉模型的先验知识，以零样本的方式实现单张图像到3D场景重建及纹理编辑。 ZeroScene从输入图像中提取对象级别的2D分割和深度信息，以此推断出场景中的空间关系。随后，通过联合优化点云的3D与2D投影损失来更新物体姿态，从而精确地对齐场景，并最终构建一个包含前景与背景且连贯完整的3D场景。 此外，ZeroScene还支持场景中对象的纹理编辑。通过对扩散模型施加约束并引入基于蒙版引导的渐进式图像生成策略，我们能够有效保持多视角下的纹理一致性，并通过物理基础渲染（PBR）材质估计进一步提升渲染结果的真实感。 实验结果显示，我们的框架不仅保证了生成资产的几何和外观准确性，还忠实还原场景布局，并产生与文本提示高度一致且细节丰富的纹理。
URL
https://arxiv.org/abs/2509.23607
