Abstract
3D Gaussians have recently emerged as an efficient representation for novel view synthesis. This work studies its editability with a particular focus on the inpainting task, which aims to supplement an incomplete set of 3D Gaussians with additional points for visually harmonious rendering. Compared to 2D inpainting, the crux of inpainting 3D Gaussians is to figure out the rendering-relevant properties of the introduced points, whose optimization largely benefits from their initial 3D positions. To this end, we propose to guide the point initialization with an image-conditioned depth completion model, which learns to directly restore the depth map based on the observed image. Such a design allows our model to fill in depth values at an aligned scale with the original depth, and also to harness strong generalizability from largescale diffusion prior. Thanks to the more accurate depth completion, our approach, dubbed InFusion, surpasses existing alternatives with sufficiently better fidelity and efficiency under various complex scenarios. We further demonstrate the effectiveness of InFusion with several practical applications, such as inpainting with user-specific texture or with novel object insertion.
Abstract (translated)
最近，3D高斯分布作为一种有效的新的视图合成表示方法而 emergence。本文重点研究了在修复任务上的可编辑性，该任务旨在通过为3D Gaussians增加视觉上和谐的数据点来补充不完整的3D Gaussians。与2D修复相比，修复3D Gaussians的关键在于确定引入的点的渲染相关性质，这些优化主要受益于它们初始的3D位置。为此，我们提出了一种基于图像条件完成模型的点初始化方法，该模型能够根据观察到的图像直接恢复深度图。这样的设计允许我们的模型在与原始深度对齐的尺度上填充深度值，并且还能从大面积扩散先验中充分利用强大的泛化能力。由于更准确的深度完成，我们的方法（被称为InFusion）在各种复杂场景下具有足够好的保真度和效率超越了现有的替代方法。我们进一步通过几个实际应用展示了InFusion的有效性，例如使用用户特定纹理进行修复或通过新颖物体插入进行修复。
URL
https://arxiv.org/abs/2404.11613
