Abstract
3D Gaussian Splatting (3DGS) has emerged as a powerful and efficient 3D representation for novel view synthesis. This paper extends 3DGS capabilities to inpainting, where masked objects in a scene are replaced with new contents that blend seamlessly with the surroundings. Unlike 2D image inpainting, 3D Gaussian inpainting (3DGI) is challenging in effectively leveraging complementary visual and semantic cues from multiple input views, as occluded areas in one view may be visible in others. To address this, we propose a method that measures the visibility uncertainties of 3D points across different input views and uses them to guide 3DGI in utilizing complementary visual cues. We also employ uncertainties to learn a semantic concept of scene without the masked object and use a diffusion model to fill masked objects in input images based on the learned concept. Finally, we build a novel 3DGI framework, VISTA, by integrating VISibility-uncerTainty-guided 3DGI with scene conceptuAl learning. VISTA generates high-quality 3DGS models capable of synthesizing artifact-free and naturally inpainted novel views. Furthermore, our approach extends to handling dynamic distractors arising from temporal object changes, enhancing its versatility in diverse scene reconstruction scenarios. We demonstrate the superior performance of our method over state-of-the-art techniques using two challenging datasets: the SPIn-NeRF dataset, featuring 10 diverse static 3D inpainting scenes, and an underwater 3D inpainting dataset derived from UTB180, including fast-moving fish as inpainting targets.
Abstract (translated)
3D高斯点云（3D Gaussian Splatting，3DGS）作为一种强大的、高效的三维表示方法，在新颖视图合成中崭露头角。本文将3DGS的能力扩展到图像修复领域，即替换场景中被遮挡对象为与周围环境无缝融合的新内容。不同于2D图像修复，3D高斯点云修复（3DGI）在有效利用来自多个输入视角的互补视觉和语义线索方面更具挑战性，因为在一个视角下被遮挡的区域可能在其他视图中可见。为此，我们提出了一种方法来测量不同输入视角之间三维点的可视不确定性，并使用这些信息指导3DGI充分利用互补的视觉提示。 同时，利用这种不确定性的特性，我们还开发了一个学习场景概念（去除遮罩物体后）的方法，并通过扩散模型根据所学的概念填充输入图像中的被遮掩对象。最后，我们构建了一个新的3DGI框架VISTA，该框架将基于可视不确定性引导的3DGI与场景概念化学习相结合。 VISTA能够生成高质量的3DGS模型，这些模型可以合成无伪影且自然修复的新视图。此外，我们的方法还扩展到了处理动态干扰物的问题，这些问题来源于物体随时间的变化，这增强了其在各种场景重建情况下的适应性。我们通过两个具有挑战性的数据集验证了该方法相对于现有技术的优越性能：一个是在10个多样化静态3D修复场景中进行测试的SPIn-NeRF数据集，另一个是从UTB180数据集中衍生出的水下3D修复数据集，其中包括快速移动的鱼作为修复目标。
URL
https://arxiv.org/abs/2504.17815
