Abstract
Adverse climate conditions pose significant challenges for autonomous systems, demanding reliable perception and decision-making across diverse environments. To better simulate these conditions, physically-based NeRF rendering methods have been explored for their ability to generate realistic scene representations. However, these methods suffer from slow rendering speeds and long preprocessing times, making them impractical for real-time testing and user interaction. This paper presents ClimateGS, a novel framework integrating 3D Gaussian representations with physical simulation to enable real-time climate effects rendering. The novelty of this work is threefold: 1) developing a linear transformation for 3D Gaussian photorealistic style transfer, enabling direct modification of spherical harmonics across bands for efficient and consistent style adaptation; 2) developing a joint training strategy for 3D style transfer, combining supervised and self-supervised learning to accelerate convergence while preserving original scene details; 3) developing a real-time rendering method for climate simulation, integrating physics-based effects with 3D Gaussian to achieve efficient and realistic rendering. We evaluate ClimateGS on MipNeRF360 and Tanks and Temples, demonstrating real-time rendering with comparable or superior visual quality to SOTA 2D/3D methods, making it suitable for interactive applications.
Abstract (translated)
恶劣的气候条件对自主系统构成了重大挑战，需要在各种环境中实现可靠感知和决策。为了更好地模拟这些条件，物理基础的NeRF渲染方法因其能够生成逼真的场景表示而受到探索。然而，这些方法存在渲染速度慢、预处理时间长的问题，这使得它们不适合实时测试和用户交互。本文介绍了ClimateGS，这是一个新的框架，它将3D高斯表示与物理仿真相结合，以实现气候效应的实时渲染。这项工作的创新性体现在三个方面： 1. **开发了用于3D高斯真实感风格迁移的线性变换**：通过直接修改各频带内的球谐函数（Spherical Harmonics），使高效的、一致的样式适应成为可能。 2. **提出了一种结合监督学习和自监督学习的联合训练策略，以加速收敛同时保持原始场景细节**。这种方法应用于3D风格迁移中。 3. **开发了一种用于气候模拟的实时渲染方法**，将基于物理的效果与3D高斯相结合，实现高效且逼真的渲染效果。 我们已经在MipNeRF360和Tanks and Temples数据集上评估了ClimateGS，并展示了其实时渲染能力，在视觉质量方面可与最新的2D/3D方法相媲美或超越。这些特性使其非常适合于互动应用。
URL
https://arxiv.org/abs/2503.14845
