Abstract
We consider the problem of physically-based inverse rendering using 3D Gaussian Splatting (3DGS) representations. While recent 3DGS methods have achieved remarkable results in novel view synthesis (NVS), accurately capturing high-fidelity geometry, physically interpretable materials and lighting remains challenging, as it requires precise geometry modeling to provide accurate surface normals, along with physically-based rendering (PBR) techniques to ensure correct material and lighting disentanglement. Previous 3DGS methods resort to approximating surface normals, but often struggle with noisy local geometry, leading to inaccurate normal estimation and suboptimal material-lighting decomposition. In this paper, we introduce GeoSplatting, a novel hybrid representation that augments 3DGS with explicit geometric guidance and differentiable PBR equations. Specifically, we bridge isosurface and 3DGS together, where we first extract isosurface mesh from a scalar field, then convert it into 3DGS points and formulate PBR equations for them in a fully differentiable manner. In GeoSplatting, 3DGS is grounded on the mesh geometry, enabling precise surface normal modeling, which facilitates the use of PBR frameworks for material decomposition. This approach further maintains the efficiency and quality of NVS from 3DGS while ensuring accurate geometry from the isosurface. Comprehensive evaluations across diverse datasets demonstrate the superiority of GeoSplatting, consistently outperforming existing methods both quantitatively and qualitatively.
Abstract (translated)
我们考虑使用三维高斯散射（3DGS）表示进行基于物理的逆渲染问题。尽管最近的3DGS方法在新颖视图合成（NVS）方面取得了显著成果，能够准确捕捉高质量几何形状、具有物理可解释性的材料和照明仍然是一个挑战。这需要精确的几何建模来提供准确的表面法线，并结合基于物理的渲染（PBR）技术以确保正确的材质和光照分离。之前的3DGS方法依赖于近似表面法线，但常常因局部几何形状嘈杂而导致法线估计不准确以及材料-照明分解次优。本文中，我们介绍了GeoSplatting，这是一种新型混合表示，通过显式的几何指导和可微分的PBR方程增强了3DGS。具体来说，我们将等值面与3DGS结合起来，在这个过程中首先从标量场提取等值面网格，然后将其转换为3DGS点并以完全可微的方式为其建立PBR方程。在GeoSplatting中，3DGS基于网格几何形状，能够进行精确的表面法线建模，从而促进使用PBR框架来分解材质。这种方法不仅保持了3DGS在NVS方面的效率和质量，还确保了从等值面获取准确的几何结构。广泛的实验评估表明，在多样化的数据集上，GeoSplatting表现出色，无论是定量还是定性指标均优于现有方法。
URL
https://arxiv.org/abs/2410.24204
