Abstract
Neural reconstruction approaches are rapidly emerging as the preferred representation for 3D scenes, but their limited editability is still posing a challenge. In this work, we propose an approach for 3D scene inpainting -- the task of coherently replacing parts of the reconstructed scene with desired content. Scene inpainting is an inherently ill-posed task as there exist many solutions that plausibly replace the missing content. A good inpainting method should therefore not only enable high-quality synthesis but also a high degree of control. Based on this observation, we focus on enabling explicit control over the inpainted content and leverage a reference image as an efficient means to achieve this goal. Specifically, we introduce RefFusion, a novel 3D inpainting method based on a multi-scale personalization of an image inpainting diffusion model to the given reference view. The personalization effectively adapts the prior distribution to the target scene, resulting in a lower variance of score distillation objective and hence significantly sharper details. Our framework achieves state-of-the-art results for object removal while maintaining high controllability. We further demonstrate the generality of our formulation on other downstream tasks such as object insertion, scene outpainting, and sparse view reconstruction.
Abstract (translated)
神经重建方法正迅速成为3D场景的首选表示方法，但它们的可编辑性仍然存在挑战。在这项工作中，我们提出了一个3D场景修复方法--在修复场景中部分内容替换所需内容。场景修复是一个本质上有问题的任务，因为存在许多合理地替换缺失内容的解决方案。因此，一个好的修复方法应该不仅能够实现高质量合成，还应该具有高程度的控制力。基于这一观察，我们专注于实现对修复内容的显式控制，并利用参考图像作为实现这一目标的有效手段。具体来说，我们引入了RefFusion，一种基于图像修复扩散模型多尺度自适应的3D修复方法。自适应有效地将先验分布适应目标场景，导致评分蒸馏目标变低，因此显著地更清晰地突出细节。我们的框架在物体去除的同时保持高可控制性。我们还进一步证明了我们在其他下游任务上的通用性，例如物体插入、场景修复和稀疏视图重建。
URL
https://arxiv.org/abs/2404.10765
