Abstract
Recovering 4D from monocular video, which jointly estimates dynamic geometry and camera poses, is an inevitably challenging problem. While recent pointmap-based 3D reconstruction methods (e.g., DUSt3R) have made great progress in reconstructing static scenes, directly applying them to dynamic scenes leads to inaccurate results. This discrepancy arises because moving objects violate multi-view geometric constraints, disrupting the reconstruction. To address this, we introduce C4D, a framework that leverages temporal Correspondences to extend existing 3D reconstruction formulation to 4D. Specifically, apart from predicting pointmaps, C4D captures two types of correspondences: short-term optical flow and long-term point tracking. We train a dynamic-aware point tracker that provides additional mobility information, facilitating the estimation of motion masks to separate moving elements from the static background, thus offering more reliable guidance for dynamic scenes. Furthermore, we introduce a set of dynamic scene optimization objectives to recover per-frame 3D geometry and camera parameters. Simultaneously, the correspondences lift 2D trajectories into smooth 3D trajectories, enabling fully integrated 4D reconstruction. Experiments show that our framework achieves complete 4D recovery and demonstrates strong performance across multiple downstream tasks, including depth estimation, camera pose estimation, and point tracking. Project Page: this https URL
Abstract (translated)
从单目视频中恢复四维（4D）信息，即同时估计动态几何和相机姿态，是一项不可避免的挑战。尽管最近基于点图的三维重建方法（如DUSt3R）在静态场景重建方面取得了巨大进展，但直接将其应用于动态场景会导致不准确的结果。这种差异出现的原因是因为移动物体违反了多视角几何约束，破坏了重建过程。为了解决这个问题，我们引入了一个名为C4D的新框架，该框架利用时间对应关系来扩展现有的三维重建方法以实现四维重建。 具体来说，在预测点图之外，C4D捕捉到了两种类型的对应关系：短期光流和长期点跟踪。我们训练了一种动态感知的点追踪器，提供额外的移动信息，便于估算运动掩模，从而将移动元素与静态背景分离，为动态场景提供了更可靠的指导。此外，我们引入了一系列针对动态场景优化的目标函数，以恢复每帧的三维几何结构和相机参数。 同时，这些对应关系将二维轨迹提升为平滑的三维轨迹，使完全集成的四维重建成为可能。实验表明，我们的框架实现了完整的4D恢复，并在包括深度估计、相机姿态估计和点跟踪在内的多个下游任务中表现出强大的性能。项目主页：[此链接](https://this-url.com/)（请将"this https URL"替换为实际项目页面链接）。
URL
https://arxiv.org/abs/2510.14960
