Abstract
Recently, 3D Gaussian Splatting, as a novel 3D representation, has garnered attention for its fast rendering speed and high rendering quality. However, this comes with high memory consumption, e.g., a well-trained Gaussian field may utilize three million Gaussian primitives and over 700 MB of memory. We credit this high memory footprint to the lack of consideration for the relationship between primitives. In this paper, we propose a memory-efficient Gaussian field named SUNDAE with spectral pruning and neural compensation. On one hand, we construct a graph on the set of Gaussian primitives to model their relationship and design a spectral down-sampling module to prune out primitives while preserving desired signals. On the other hand, to compensate for the quality loss of pruning Gaussians, we exploit a lightweight neural network head to mix splatted features, which effectively compensates for quality losses while capturing the relationship between primitives in its weights. We demonstrate the performance of SUNDAE with extensive results. For example, SUNDAE can achieve 26.80 PSNR at 145 FPS using 104 MB memory while the vanilla Gaussian splatting algorithm achieves 25.60 PSNR at 160 FPS using 523 MB memory, on the Mip-NeRF360 dataset. Codes are publicly available at this https URL.
Abstract (translated)
最近，3D高斯展平（3D Gausian Splatting）作为一种新颖的3D表示方式，因其快速的渲染速度和高的渲染质量而引起了关注。然而，这也带来了高内存消耗，例如，经过良好训练的高斯场可能会使用300万高斯原型，需要超过700MB的内存。我们将高内存消耗归因于对原语之间关系的缺乏考虑。在本文中，我们提出了一个内存高效的Gausian场，名为SUNDAE，通过谱滤波和神经补偿实现。一方面，我们在高斯原子的集合上构建一个图，建模它们之间的关系，并设计了一个用于保留所需信号的高斯下采样模块，另一方面，为了弥补修剪高斯场的质量损失，我们利用轻量级的神经网络头混合展平特征，有效地补偿质量损失并捕获原语之间的关系。我们通过广泛的实验结果证明了SUNDAE的性能。例如，SUNDAE在145 FPS时可以达到26.80 PSNR，而原神和高斯展平算法在160 FPS时可以达到25.60 PSNR，在Mip-NeRF360数据集上。代码 publicly available at this https URL.
URL
https://arxiv.org/abs/2405.00676
