Abstract
High-fidelity 3D video reconstruction is essential for enabling real-time rendering of dynamic scenes with realistic motion in virtual and augmented reality (VR/AR). The deformation field paradigm of 3D Gaussian splatting has achieved near-photorealistic results in video reconstruction due to the great representation capability of deep deformation networks. However, in videos with complex motion and significant scale variations, deformation networks often overfit to irregular Gaussian trajectories, leading to suboptimal visual quality. Moreover, the gradient-based densification strategy designed for static scene reconstruction proves inadequate to address the absence of dynamic content. In light of these challenges, we propose a flow-empowered velocity field modeling scheme tailored for Gaussian video reconstruction, dubbed FlowGaussian-VR. It consists of two core components: a velocity field rendering (VFR) pipeline which enables optical flow-based optimization, and a flow-assisted adaptive densification (FAD) strategy that adjusts the number and size of Gaussians in dynamic regions. We validate our model's effectiveness on multi-view dynamic reconstruction and novel view synthesis with multiple real-world datasets containing challenging motion scenarios, demonstrating not only notable visual improvements (over 2.5 dB gain in PSNR) and less blurry artifacts in dynamic textures, but also regularized and trackable per-Gaussian trajectories.
Abstract (translated)
高保真3D视频重建对于在虚拟现实（VR）和增强现实（AR）中实时渲染具有真实运动的动态场景至关重要。基于三维高斯点云的变形场范式，由于深度变形网络的强大表征能力，在视频重建方面已经实现了接近照片级的真实效果。然而，在复杂运动和显著尺度变化的视频中，变形网络往往会过度拟合不规则的高斯轨迹，导致视觉质量下降。此外，用于静态场景重建的基于梯度的稠密化策略在处理缺乏动态内容的问题上显得不足。 针对这些挑战，我们提出了一种专为高斯视频重建设计的速度场建模方案——FlowGaussian-VR。该方案包含两个核心组件：速度场渲染（VFR）管线，它支持以光流为基础进行优化；以及一种受流动辅助的自适应稠密化（FAD）策略，它可以调整动态区域中高斯的数量和大小。 我们使用多个包含挑战性运动场景的真实世界数据集，在多视角动态重建和新视点合成上验证了模型的有效性。结果显示，我们的方法不仅在视觉质量上有显著提升（PSNR值提高了超过2.5dB），而且减少了动态纹理中的模糊伪影，并确保每颗高斯的轨迹更加规律且可追踪。
URL
https://arxiv.org/abs/2507.23704
