Abstract
Recent breakthroughs in text-guided image generation have significantly advanced the field of 3D generation. While generating a single high-quality 3D object is now feasible, generating multiple objects with reasonable interactions within a 3D space, a.k.a. compositional 3D generation, presents substantial challenges. This paper introduces CompGS, a novel generative framework that employs 3D Gaussian Splatting (GS) for efficient, compositional text-to-3D content generation. To achieve this goal, two core designs are proposed: (1) 3D Gaussians Initialization with 2D compositionality: We transfer the well-established 2D compositionality to initialize the Gaussian parameters on an entity-by-entity basis, ensuring both consistent 3D priors for each entity and reasonable interactions among multiple entities; (2) Dynamic Optimization: We propose a dynamic strategy to optimize 3D Gaussians using Score Distillation Sampling (SDS) loss. CompGS first automatically decomposes 3D Gaussians into distinct entity parts, enabling optimization at both the entity and composition levels. Additionally, CompGS optimizes across objects of varying scales by dynamically adjusting the spatial parameters of each entity, enhancing the generation of fine-grained details, particularly in smaller entities. Qualitative comparisons and quantitative evaluations on T3Bench demonstrate the effectiveness of CompGS in generating compositional 3D objects with superior image quality and semantic alignment over existing methods. CompGS can also be easily extended to controllable 3D editing, facilitating scene generation. We hope CompGS will provide new insights to the compositional 3D generation. Project page: this https URL.
Abstract (translated)
近期，文本引导的图像生成领域取得了显著突破，大大推动了三维生成技术的发展。虽然现在生成单个高质量的3D对象已经成为可能，但在三维空间内生成具有合理互动的多个对象，即组合式3D生成，仍面临巨大挑战。本文介绍了一种新型生成框架CompGS，它利用3D高斯喷绘（GS）实现高效的、基于文本到3D内容的组合生成。为达到这一目标，提出了两个核心设计：(1) 基于2D组合性的3D高斯初始化：我们借鉴成熟的2D组合性原理，在实体级别上初始化高斯参数，确保每个实体都有一致的3D先验，并且多个实体之间存在合理的互动；(2) 动态优化：提出了一种动态策略，使用得分蒸馏采样（SDS）损失来优化3D高斯。CompGS首先自动将3D高斯分解成不同的实体部分，从而能够在实体和组合两个层面上进行优化。此外，通过动态调整每个实体的空间参数，CompGS可以跨不同尺度的对象进行优化，增强细节生成能力，特别是在较小的实体上。T3Bench上的定性和定量评估表明，与现有方法相比，CompGS在生成具有优越图像质量和语义对齐的组合式3D对象方面表现出色。CompGS还能够轻松扩展到可控的3D编辑，促进场景生成。我们希望CompGS能为组合式3D生成提供新的见解。项目页面：[此处链接]
URL
https://arxiv.org/abs/2410.20723
