Abstract
Generalizable 3D Gaussian splitting (3DGS) can reconstruct new scenes from sparse-view observations in a feed-forward inference manner, eliminating the need for scene-specific retraining required in conventional 3DGS. However, existing methods rely heavily on epipolar priors, which can be unreliable in complex realworld scenes, particularly in non-overlapping and occluded regions. In this paper, we propose eFreeSplat, an efficient feed-forward 3DGS-based model for generalizable novel view synthesis that operates independently of epipolar line constraints. To enhance multiview feature extraction with 3D perception, we employ a selfsupervised Vision Transformer (ViT) with cross-view completion pre-training on large-scale datasets. Additionally, we introduce an Iterative Cross-view Gaussians Alignment method to ensure consistent depth scales across different views. Our eFreeSplat represents an innovative approach for generalizable novel view synthesis. Different from the existing pure geometry-free methods, eFreeSplat focuses more on achieving epipolar-free feature matching and encoding by providing 3D priors through cross-view pretraining. We evaluate eFreeSplat on wide-baseline novel view synthesis tasks using the RealEstate10K and ACID datasets. Extensive experiments demonstrate that eFreeSplat surpasses state-of-the-art baselines that rely on epipolar priors, achieving superior geometry reconstruction and novel view synthesis quality. Project page: this https URL.
Abstract (translated)
通用化的3D高斯分裂（3DGS）能够以前向推理的方式，从稀疏视角观察中重建新的场景，消除了传统3DGS所需针对特定场景重新训练的需要。然而，现有的方法很大程度上依赖于极线先验，在复杂的真实世界场景中，特别是在非重叠和遮挡区域，这些先验可能不可靠。在本文中，我们提出了eFreeSplat，这是一种高效的前向推理3DGS模型，适用于通用化的新型视图合成，并且独立于极线约束运行。为了增强具有三维感知的多视角特征提取，我们使用了带有大规模数据集上跨视角补全预训练的自监督视觉变换器（ViT）。此外，我们引入了一种迭代交叉视角高斯对齐方法来确保不同视图之间的深度比例一致。我们的eFreeSplat代表了一种创新的方法来进行通用化的新型视图合成。与现有的纯几何无关的方法不同，eFreeSplat更侧重于通过跨视角预训练提供的三维先验实现无极线的特征匹配和编码。我们在宽基线新型视图合成任务上使用RealEstate10K和ACID数据集对eFreeSplat进行了评估。大量的实验表明，与依赖于极线先验的最先进的基准相比，eFreeSplat在几何重建和新型视图合成功能方面都表现出色。项目页面：此 https URL。
URL
https://arxiv.org/abs/2410.22817
