Abstract
Reconstructing controllable Gaussian splats from monocular video is a challenging task due to its inherently insufficient constraints. Widely adopted approaches supervise complex interactions with additional masks and control signal annotations, limiting their real-world applications. In this paper, we propose an annotation guidance-free method, dubbed FreeGaussian, that mathematically derives dynamic Gaussian motion from optical flow and camera motion using novel dynamic Gaussian constraints. By establishing a connection between 2D flows and 3D Gaussian dynamic control, our method enables self-supervised optimization and continuity of dynamic Gaussian motions from flow priors. Furthermore, we introduce a 3D spherical vector controlling scheme, which represents the state with a 3D Gaussian trajectory, thereby eliminating the need for complex 1D control signal calculations and simplifying controllable Gaussian modeling. Quantitative and qualitative evaluations on extensive experiments demonstrate the state-of-the-art visual performance and control capability of our method. Project page: this https URL.
Abstract (translated)
从单目视频中重构可控制的高斯斑点是一个具有挑战性的任务，因为其内在的约束条件不足。广泛采用的方法通过添加额外的掩码和控制信号注释来监督复杂交互，这限制了它们在现实世界中的应用。在这篇论文中，我们提出了一种无需标注指导的方法，称为FreeGaussian，它利用新颖的动态高斯约束从光流和相机运动中数学推导出动态高斯运动。通过建立2D流与3D高斯动态控制之间的联系，我们的方法实现了自我监督优化，并使动态高斯运动具有连续性。此外，我们引入了一种三维球形向量控制方案，该方案用一个3D高斯轨迹来表示状态，从而消除了对复杂的一维控制信号计算的需求，并简化了可控制的高斯模型建立过程。大量的实验定量和定性的评估表明，我们的方法具有最先进的视觉性能和控制能力。项目页面：这个 https URL。
URL
https://arxiv.org/abs/2410.22070
