Abstract
We introduce a diffusion-based framework that performs aligned novel view image and geometry generation via a warping-and-inpainting methodology. Unlike prior methods that require dense posed images or pose-embedded generative models limited to in-domain views, our method leverages off-the-shelf geometry predictors to predict partial geometries viewed from reference images, and formulates novel-view synthesis as an inpainting task for both image and geometry. To ensure accurate alignment between generated images and geometry, we propose cross-modal attention distillation, where attention maps from the image diffusion branch are injected into a parallel geometry diffusion branch during both training and inference. This multi-task approach achieves synergistic effects, facilitating geometrically robust image synthesis as well as well-defined geometry prediction. We further introduce proximity-based mesh conditioning to integrate depth and normal cues, interpolating between point cloud and filtering erroneously predicted geometry from influencing the generation process. Empirically, our method achieves high-fidelity extrapolative view synthesis on both image and geometry across a range of unseen scenes, delivers competitive reconstruction quality under interpolation settings, and produces geometrically aligned colored point clouds for comprehensive 3D completion. Project page is available at this https URL.
Abstract (translated)
我们提出了一种基于扩散的框架，通过扭曲和修复的方法生成与新视角图像和几何形状对齐的新视图。与之前需要密集姿态图像或局限于领域内视图的嵌入姿势的生成模型的方法不同，我们的方法利用现成的几何预测器来预测从参考图像观察到的部分几何，并将新视图合成视为同时针对图像和几何的修复任务。为了确保生成的图像和几何之间精确对齐，我们提出了跨模态注意力蒸馏技术，在训练和推理过程中，将来自图像扩散分支的关注图注入平行的几何扩散分支中。这种多任务方法实现了协同效应，有助于实现几何鲁棒的图像合成以及明确定义的几何预测。 此外，为了整合深度和法线提示，我们将基于邻近性的网格条件引入该框架，通过在点云之间进行插值，并过滤掉错误预测的几何形状对生成过程的影响。实验证明，在各种未见过的场景中，我们的方法能够在图像和几何上实现高保真的外推视图合成；在内插设置下提供具有竞争力的重建质量；并且可以产生与颜色匹配的点云以进行全面的3D完成。 项目页面可在[此处](https://this-url.com)找到。
URL
https://arxiv.org/abs/2506.11924
