Abstract
Reconstructing and editing 3D objects and scenes both play crucial roles in computer graphics and computer vision. Neural radiance fields (NeRFs) can achieve realistic reconstruction and editing results but suffer from inefficiency in rendering. Gaussian splatting significantly accelerates rendering by rasterizing Gaussian ellipsoids. However, Gaussian splatting utilizes a single Spherical Harmonic (SH) function to model both texture and lighting, limiting independent editing capabilities of these components. Recently, attempts have been made to decouple texture and lighting with the Gaussian splatting representation but may fail to produce plausible geometry and decomposition results on reflective scenes. Additionally, the forward shading technique they employ introduces noticeable blending artifacts during relighting, as the geometry attributes of Gaussians are optimized under the original illumination and may not be suitable for novel lighting conditions. To address these issues, we introduce DeferredGS, a method for decoupling and editing the Gaussian splatting representation using deferred shading. To achieve successful decoupling, we model the illumination with a learnable environment map and define additional attributes such as texture parameters and normal direction on Gaussians, where the normal is distilled from a jointly trained signed distance function. More importantly, we apply deferred shading, resulting in more realistic relighting effects compared to previous methods. Both qualitative and quantitative experiments demonstrate the superior performance of DeferredGS in novel view synthesis and editing tasks.
Abstract (translated)
重构和编辑3D对象和场景在计算机图形学和计算机视觉中扮演着关键角色。神经元辐射场（NeRFs）可以实现逼真的重建和编辑结果，但渲染过程中效率较低。高斯平铺显著加速了渲染，通过将高斯椭球体进行平面化。然而，高斯平铺仅使用单个球形谐波（SH）函数来建模纹理和光照，这限制了这些组件的独立编辑能力。最近，人们尝试将纹理和光照与高斯平铺表示解耦，但可能在反射场景上产生不合理的几何和分解结果。此外，他们采用的方法在重新光照过程中引入了明显的混合伪影，因为原始光照下高斯粒子的几何属性被优化，可能不适用于新的照明条件。为了解决这些问题，我们引入了DeferredGS，一种使用延迟光照解耦和编辑高斯平铺表示的方法。为了实现成功的解耦，我们用可学习的环境图建模光照，并在高斯粒子上定义了纹理参数和法线方向等附加属性，其中法线是从联合训练的签名距离函数中蒸馏的。更重要的是，我们应用了延迟光照，导致与以前方法相比更真实的重新光照效果。 both qualitative and quantitative experiments demonstrate the superior performance of DeferredGS in novel view synthesis and editing tasks.
URL
https://arxiv.org/abs/2404.09412
