Abstract
Recent advances in 3D scene generation produce visually appealing output, but current representations hinder artists' workflows that require modifiable 3D textured mesh scenes for visual effects and game development. Despite significant advances, current textured mesh scene reconstruction methods are far from artist ready, suffering from incorrect object decomposition, inaccurate spatial relationships, and missing backgrounds. We present 3D-RE-GEN, a compositional framework that reconstructs a single image into textured 3D objects and a background. We show that combining state of the art models from specific domains achieves state of the art scene reconstruction performance, addressing artists' requirements. Our reconstruction pipeline integrates models for asset detection, reconstruction, and placement, pushing certain models beyond their originally intended domains. Obtaining occluded objects is treated as an image editing task with generative models to infer and reconstruct with scene level reasoning under consistent lighting and geometry. Unlike current methods, 3D-RE-GEN generates a comprehensive background that spatially constrains objects during optimization and provides a foundation for realistic lighting and simulation tasks in visual effects and games. To obtain physically realistic layouts, we employ a novel 4-DoF differentiable optimization that aligns reconstructed objects with the estimated ground plane. 3D-RE-GEN~achieves state of the art performance in single image 3D scene reconstruction, producing coherent, modifiable scenes through compositional generation guided by precise camera recovery and spatial optimization.
Abstract (translated)
最近在三维场景生成领域的进展产生了视觉上吸引人的输出，但当前的表示方法阻碍了艺术家的工作流程，这些工作需要可修改的具有纹理的三维网格场景用于视觉效果和游戏开发。尽管已经取得了重大进步，目前的纹理网格场景重建方法仍远未达到适合艺术家使用的要求，它们在对象分解、空间关系准确性以及背景缺失方面存在不足。 我们提出了3D-RE-GEN，这是一种组合框架，可以将单张图像重构为具有纹理的三维物体和背景。研究表明，结合来自特定领域的最先进的模型可以实现最佳场景重建性能，并满足艺术家的需求。我们的重建管道集成了资产检测、重建和放置的模型，在某些情况下超越了原定的应用领域。获取被遮挡的对象被视为一项图像编辑任务，使用生成模型进行推理并在一致的光照和几何结构下重构整个场景。与当前的方法不同，3D-RE-GEN 生成全面的背景，该背景在优化过程中空间上限制物体，并为视觉效果和游戏中的现实主义照明和仿真提供基础。 为了获得物理上真实的布局，我们采用了一种新颖的四自由度可微分优化方法，将重建的对象与估计的地平面对齐。3D-RE-GEN 在单图像三维场景重构方面取得了最先进的性能，通过由精确相机恢复和空间优化引导的组合生成方法产生了连贯且可修改的场景。
URL
https://arxiv.org/abs/2512.17459
