Abstract
We present DiSR-NeRF, a diffusion-guided framework for view-consistent super-resolution (SR) NeRF. Unlike prior works, we circumvent the requirement for high-resolution (HR) reference images by leveraging existing powerful 2D super-resolution models. Nonetheless, independent SR 2D images are often inconsistent across different views. We thus propose Iterative 3D Synchronization (I3DS) to mitigate the inconsistency problem via the inherent multi-view consistency property of NeRF. Specifically, our I3DS alternates between upscaling low-resolution (LR) rendered images with diffusion models, and updating the underlying 3D representation with standard NeRF training. We further introduce Renoised Score Distillation (RSD), a novel score-distillation objective for 2D image resolution. Our RSD combines features from ancestral sampling and Score Distillation Sampling (SDS) to generate sharp images that are also LR-consistent. Qualitative and quantitative results on both synthetic and real-world datasets demonstrate that our DiSR-NeRF can achieve better results on NeRF super-resolution compared with existing works. Code and video results available at the project website.
Abstract (translated)
我们提出了DiSR-NeRF,一种扩散引导的视图一致超分辨率(SR) NeRF框架。与之前的工作不同,我们通过利用现有的强大2D超分辨率模型绕过了高分辨率(HR)参考图像的要求。然而,不同的视图中的独立SR 2D图像通常是不一致的。因此,我们提出了迭代3D同步(I3DS)来通过NeRF固有的多视图一致性特性来缓解不一致性问题。具体来说,我们的I3DS交替使用扩散模型上采样低分辨率(LR)渲染图像,并使用标准NeRF训练更新底层3D表示。我们还引入了去噪得分蒸馏(RSD)和新型的分数蒸馏采样(SDS)用于2D图像分辨率。我们的RSD结合了祖先生成度和分数蒸馏采样(SDS)的特征,生成具有清晰度的图像,并且也是LR一致的。在合成和真实世界数据集上的结果表明,与现有工作相比,我们的DiSR-NeRF在NeRF超分辨率方面可以实现更好的结果。代码和视频结果可以在项目网站上查看。
URL
https://arxiv.org/abs/2404.00874
