Abstract
Recent advances in 3D scene reconstruction enable real-time viewing in virtual and augmented reality. To support interactive operations for better immersiveness, such as moving or editing objects, 3D scene inpainting methods are proposed to repair or complete the altered geometry. However, current approaches rely on lengthy and computationally intensive optimization, making them impractical for real-time or online applications. We propose InstaInpaint, a reference-based feed-forward framework that produces 3D-scene inpainting from a 2D inpainting proposal within 0.4 seconds. We develop a self-supervised masked-finetuning strategy to enable training of our custom large reconstruction model (LRM) on the large-scale dataset. Through extensive experiments, we analyze and identify several key designs that improve generalization, textural consistency, and geometric correctness. InstaInpaint achieves a 1000x speed-up from prior methods while maintaining a state-of-the-art performance across two standard benchmarks. Moreover, we show that InstaInpaint generalizes well to flexible downstream applications such as object insertion and multi-region inpainting. More video results are available at our project page: this https URL.
Abstract (translated)
最近在3D场景重建领域的进展使得虚拟现实和增强现实中实时光学成为可能。为了支持更好的沉浸感并实现诸如移动或编辑对象的互动操作，提出了一些三维场景修复方法（即3D scene inpainting），用于修正或完成被改变的几何形状。然而，目前的方法依赖于耗时且计算密集型的优化过程，这使得它们不适合实时或在线应用。 我们提出了InstaInpaint，这是一种基于参考的前馈框架，能够从一个2D修复建议中生成3D场景修补结果，在0.4秒内完成任务。为了训练我们的自定义大型重建模型（LRM）以适应大规模数据集，我们开发了一种自我监督的掩码微调策略。通过广泛的实验，我们分析并识别了几项关键设计改进，这些改进提升了泛化能力、纹理一致性和几何正确性。 InstaInpaint相较于先前的方法，在速度上实现了1000倍的提升，并且在两个标准基准测试中保持了最先进的性能水平。此外，我们展示了InstaInpaint在诸如对象插入和多区域修补等灵活的应用场景中的良好泛化能力。更多视频结果可在我们的项目页面查看：[这个链接](https://example.com)（原文中提供的URL格式）。
URL
https://arxiv.org/abs/2506.10980
