Abstract
Reconstructing 3D objects from a single image is a long-standing challenge, especially under real-world occlusions. While recent diffusion-based view synthesis models can generate consistent novel views from a single RGB image, they generally assume fully visible inputs and fail when parts of the object are occluded. This leads to inconsistent views and degraded 3D reconstruction quality. To overcome this limitation, we propose an end-to-end framework for occlusion-aware multi-view generation. Our method directly synthesizes six structurally consistent novel views from a single partially occluded image, enabling downstream 3D reconstruction without requiring prior inpainting or manual annotations. We construct a self-supervised training pipeline using the Pix2Gestalt dataset, leveraging occluded-unoccluded image pairs and pseudo-ground-truth views to teach the model structure-aware completion and view consistency. Without modifying the original architecture, we fully fine-tune the view synthesis model to jointly learn completion and multi-view generation. Additionally, we introduce the first benchmark for occlusion-aware reconstruction, encompassing diverse occlusion levels, object categories, and mask patterns. This benchmark provides a standardized protocol for evaluating future methods under partial occlusions. Our code is available at this https URL.
Abstract (translated)
从单张图像重建3D物体长期以来一直是一个挑战，尤其是在现实世界的遮挡情况下。虽然最近基于扩散的视图合成模型能够从单一RGB图像生成一致的新视角，但它们通常假设输入是完全可见的，并且在对象的部分被遮挡时会失效。这会导致不一致的视图和降低的3D重建质量。为了解决这一限制，我们提出了一种端到端框架，用于感知遮挡的多视角生成。我们的方法直接从单张部分被遮挡的图像中合成六个结构上一致的新视角，从而在不需要预先进行图像修复或手动注释的情况下支持下游的3D重建工作。我们使用Pix2Gestalt数据集构建了一个自我监督训练管道，利用有无遮挡的图像对和伪真实视图来教导模型感知结构完成及视图一致性。在不修改原始架构的情况下，我们将视角合成模型完全微调以同时学习补全和多视角生成任务。 此外，我们还推出了首个用于感知遮挡重建的基准测试，涵盖了多样化的遮挡级别、物体类别以及掩码模式。这一基准为评估未来方法在部分被遮挡情况下的表现提供了标准化协议。我们的代码可在此链接获取（原链接已提供）。
URL
https://arxiv.org/abs/2506.21544
