Abstract
Recent advancements in 3D reconstruction technologies have paved the way for high-quality and real-time rendering of complex 3D scenes. Despite these achievements, a notable challenge persists: it is difficult to precisely reconstruct specific objects from large scenes. Current scene reconstruction techniques frequently result in the loss of object detail textures and are unable to reconstruct object portions that are occluded or unseen in views. To address this challenge, we delve into the meticulous 3D reconstruction of specific objects within large scenes and propose a framework termed OMEGAS: Object Mesh Extraction from Large Scenes Guided by GAussian Segmentation. OMEGAS employs a multi-step approach, grounded in several excellent off-the-shelf methodologies. Specifically, initially, we utilize the Segment Anything Model (SAM) to guide the segmentation of 3D Gaussian Splatting (3DGS), thereby creating a basic 3DGS model of the target object. Then, we leverage large-scale diffusion priors to further refine the details of the 3DGS model, especially aimed at addressing invisible or occluded object portions from the original scene views. Subsequently, by re-rendering the 3DGS model onto the scene views, we achieve accurate object segmentation and effectively remove the background. Finally, these target-only images are used to improve the 3DGS model further and extract the definitive 3D object mesh by the SuGaR model. In various scenarios, our experiments demonstrate that OMEGAS significantly surpasses existing scene reconstruction methods. Our project page is at: this https URL
Abstract (translated)
近年来，三维重建技术的发展为高质量和实时渲染复杂的3D场景奠定了基础。然而，一个显著的挑战仍然存在：很难精确从大型场景中重建特定对象。当前的场景重建技术通常会导致丢失物体细节纹理，并且无法从视图中重建被遮挡或未见到的物体部分。为了应对这个挑战，我们深入研究了大场景中具体对象的3D重建，并提出了一个名为OMEGAS的框架：基于Gaussian分割的大型场景引导对象网格提取。OMEGAS采用了一种多步骤方法，基于几种出色的非处方方法。具体来说，首先，我们使用SAM引导3D高斯平铺（3DGS）的分割，从而创建了目标对象的初步3DGS模型。然后，我们利用大型扩散 prior进一步优化3DGS模型的细节，特别关注解决原始场景视图中看不见或被遮挡的物体部分。接下来，通过将3DGS模型重新渲染到场景视图中，我们实现了精确的物体分割，并有效地去除了背景。最后，这些目标仅图像被用于进一步改进3DGS模型，并使用SuGaR模型提取了最终的3D物体网格。在各种场景中，我们的实验表明，OMEGAS显著超越了现有的场景重建方法。我们的项目页面是：https:// this URL
URL
https://arxiv.org/abs/2404.15891
