Abstract
Developing a unified pipeline that enables users to remove, re-texture, or replace objects in a versatile manner is crucial for text-guided 3D inpainting. However, there are still challenges in performing multiple 3D inpainting tasks within a unified framework: 1) Single reference inpainting methods lack robustness when dealing with views that are far from the reference view. 2) Appearance inconsistency arises when independently inpainting multi-view images with 2D diffusion priors; 3) Geometry inconsistency limits performance when there are significant geometric changes in the inpainting regions. To tackle these challenges, we introduce DiGA3D, a novel and versatile 3D inpainting pipeline that leverages diffusion models to propagate consistent appearance and geometry in a coarse-to-fine manner. First, DiGA3D develops a robust strategy for selecting multiple reference views to reduce errors during propagation. Next, DiGA3D designs an Attention Feature Propagation (AFP) mechanism that propagates attention features from the selected reference views to other views via diffusion models to maintain appearance consistency. Furthermore, DiGA3D introduces a Texture-Geometry Score Distillation Sampling (TG-SDS) loss to further improve the geometric consistency of inpainted 3D scenes. Extensive experiments on multiple 3D inpainting tasks demonstrate the effectiveness of our method. The project page is available at this https URL.
Abstract (translated)
开发一个统一的管道，使用户能够以灵活的方式移除、重新着色或替换对象，对于文本引导的3D修复（inpainting）至关重要。然而，在单一框架内执行多个3D修复任务仍然存在挑战： 1) 单一参考修复方法在处理远离参考视图的视角时缺乏鲁棒性。 2) 独立地使用2D扩散先验修复多视图图像会导致外观不一致。 3) 几何变化显著的情况下，几何一致性限制了修复区域的表现。 为了应对这些挑战，我们引入了DiGA3D，这是一种新颖且多功能的3D修复管道，它利用扩散模型以粗到细的方式传播一致的外观和几何结构。首先，DiGA3D开发了一种稳健的方法来选择多个参考视图，从而减少在传播过程中产生的误差。其次，DiGA3D设计了一个注意力特征传播（AFP）机制，该机制通过扩散模型将选定参考视图的注意特征传递到其他视图中以保持外观一致性。此外，DiGA3D还引入了纹理-几何评分蒸馏采样（TG-SDS）损失函数，进一步提高了修复后的3D场景中的几何一致性。 在多个3D修复任务上的广泛实验表明了我们方法的有效性。该项目页面可在此[URL]访问。请注意，在实际引用时，请确保使用正确的项目网址链接，因为上述提示中提供的“https URL”需要被具体的网页地址所替换。
URL
https://arxiv.org/abs/2507.00429
