Abstract
Current compositional image-to-3D scene generation approaches construct 3D scenes by time-consuming iterative layout optimization or inflexible joint object-layout generation. Moreover, most methods rely on limited field-of-view perspective images, hindering the creation of complete 360-degree environments. To address these limitations, we design Pano3DComposer, an efficient feed-forward framework for panoramic images. To decouple object generation from layout estimation, we propose a plug-and-play Object-World Transformation Predictor. This module converts the 3D objects generated by off-the-shelf image-to-3D models from local to world coordinates. To achieve this, we adapt the VGGT architecture to Alignment-VGGT by using target object crop, multi-view object renderings and camera parameters to predict the transformation. The predictor is trained using pseudo-geometric supervision to address the shape discrepancy between generated and ground-truth objects. For input images from unseen domains, we further introduce a Coarse-to-Fine (C2F) alignment mechanism for Pano3DComposer that iteratively refines geometric consistency with feedback of scene rendering. Our method achieves superior geometric accuracy for image/text-to-3D tasks on synthetic and real-world datasets. It can generate a high-fidelity 3D scene in approximately 20 seconds on an RTX 4090 GPU. Project page: this https URL.
Abstract (translated)
当前的图像到三维场景生成方法通过耗时的迭代布局优化或刚性的联合对象-布局生成来构建3D场景。此外，大多数方法依赖于视野有限的透视图图像，这阻碍了创建完整的360度环境。为了解决这些限制，我们设计了一种面向全景图像的有效前馈框架Pano3DComposer。为了将物体生成与布局估计解耦，我们提出了一个即插即用的对象-世界变换预测器。该模块可以将现成的图像到三维模型产生的局部坐标中的三维对象转换为世界坐标系。为此，我们将VGGT架构通过使用目标物裁剪、多视角物体渲染以及相机参数调整为Alignment-VGGT来预测转换。我们利用伪几何监督训练预测器以解决生成和地面真实物体之间的形状差异问题。 对于来自未知领域的输入图像，我们进一步为Pano3DComposer引入了一种粗到细（C2F）对齐机制，该机制通过反馈场景渲染迭代地细化了几何一致性。我们的方法在合成数据集和现实世界数据集中实现了比现有技术更优越的几何精度，在将图片或文本转换成三维模型任务上表现尤为出色。它可以在NVIDIA RTX 4090 GPU上大约20秒内生成一个高保真的3D场景。 项目页面：[这里插入链接，原文中的链接无法直接在此显示]
URL
https://arxiv.org/abs/2603.05908
