Abstract
We present SCube, a novel method for reconstructing large-scale 3D scenes (geometry, appearance, and semantics) from a sparse set of posed images. Our method encodes reconstructed scenes using a novel representation VoxSplat, which is a set of 3D Gaussians supported on a high-resolution sparse-voxel scaffold. To reconstruct a VoxSplat from images, we employ a hierarchical voxel latent diffusion model conditioned on the input images followed by a feedforward appearance prediction model. The diffusion model generates high-resolution grids progressively in a coarse-to-fine manner, and the appearance network predicts a set of Gaussians within each voxel. From as few as 3 non-overlapping input images, SCube can generate millions of Gaussians with a 1024^3 voxel grid spanning hundreds of meters in 20 seconds. Past works tackling scene reconstruction from images either rely on per-scene optimization and fail to reconstruct the scene away from input views (thus requiring dense view coverage as input) or leverage geometric priors based on low-resolution models, which produce blurry results. In contrast, SCube leverages high-resolution sparse networks and produces sharp outputs from few views. We show the superiority of SCube compared to prior art using the Waymo self-driving dataset on 3D reconstruction and demonstrate its applications, such as LiDAR simulation and text-to-scene generation.
Abstract (translated)
我们提出了SCube，这是一种全新的方法，可以从一组稀疏的已定位图像中重建大规模的3D场景（包括几何结构、外观和语义）。我们的方法使用了一种新颖的表示形式VoxSplat来编码重建后的场景，这种表示形式是一组基于高分辨率稀疏体素框架支持的三维高斯分布。为了从图像中重建一个VoxSplat，我们采用了一个分层的体素潜在扩散模型，该模型依赖于输入图像，并随后使用前馈外观预测模型进行处理。扩散模型以由粗到细的方式逐步生成高分辨率网格，而外观网络则在每个体素内预测一组高斯分布。SCube可以从少至3个不重叠的输入图像中，在20秒内生成数百万个高斯分布在1024^3的体素网格上，该网格横跨数百米。 以往处理从图像进行场景重建的工作要么依赖于针对每个场景的优化，并且无法在远离输入视图的地方重构场景（因此需要密集视图覆盖作为输入），要么基于低分辨率模型使用几何先验，这会产生模糊的结果。相比之下，SCube利用高分辨率稀疏网络，能够从少量视图中生成清晰的结果。 我们通过Waymo自动驾驶数据集上的3D重建任务展示了SCube相较于以往技术的优势，并演示了它的应用，如激光雷达模拟和文本到场景的生成。
URL
https://arxiv.org/abs/2410.20030
