Abstract
With the widespread use of virtual reality applications, 3D scene generation has become a new challenging research frontier. 3D scenes have highly complex structures and need to ensure that the output is dense, coherent, and contains all necessary structures. Many current 3D scene generation methods rely on pre-trained text-to-image diffusion models and monocular depth estimators. However, the generated scenes occupy large amounts of storage space and often lack effective regularisation methods, leading to geometric distortions. To this end, we propose BloomScene, a lightweight structured 3D Gaussian splatting for crossmodal scene generation, which creates diverse and high-quality 3D scenes from text or image inputs. Specifically, a crossmodal progressive scene generation framework is proposed to generate coherent scenes utilizing incremental point cloud reconstruction and 3D Gaussian splatting. Additionally, we propose a hierarchical depth prior-based regularization mechanism that utilizes multi-level constraints on depth accuracy and smoothness to enhance the realism and continuity of the generated scenes. Ultimately, we propose a structured context-guided compression mechanism that exploits structured hash grids to model the context of unorganized anchor attributes, which significantly eliminates structural redundancy and reduces storage overhead. Comprehensive experiments across multiple scenes demonstrate the significant potential and advantages of our framework compared with several baselines.
Abstract (translated)
随着虚拟现实应用的广泛使用，3D场景生成已经成为一个新的研究前沿。3D场景具有高度复杂的结构，并且需要确保输出是密集、连贯并且包含所有必要的结构。许多当前的3D场景生成方法依赖于预训练的文字到图像扩散模型和单目深度估计器。然而，这些方法产生的场景占用大量的存储空间，并且往往缺乏有效的正则化方法，导致几何变形。为了解决这些问题，我们提出了BloomScene，这是一种轻量级的结构化的3D高斯点着色方法，用于跨模态场景生成，可以从文本或图像输入创建多样性和高质量的3D场景。 具体而言，我们提出了一种跨模式逐步场景生成框架，利用增量点云重建和3D高斯点着色来生成连贯的场景。此外，我们还提出了基于分层深度先验的正则化机制，通过在深度准确度和平滑度上应用多层次约束，增强生成场景的真实感和连续性。最终，我们提出了一种结构化的上下文引导压缩机制，利用有组织的哈希网格来建模无序锚点属性的上下文信息，从而显著消除结构冗余并减少存储开销。 在多个场景上的综合实验表明，我们的框架与几种基准方法相比具有重要的潜力和优势。
URL
https://arxiv.org/abs/2501.10462
